2023-10-27 23:52:39,876 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure timeout: 13 mins 2023-10-27 23:52:40,098 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21 2023-10-27 23:52:40,112 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-10-27 23:52:40,113 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957, deleteOnExit=true 2023-10-27 23:52:40,113 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-10-27 23:52:40,114 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/test.cache.data in system properties and HBase conf 2023-10-27 23:52:40,114 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/hadoop.tmp.dir in system properties and HBase conf 2023-10-27 23:52:40,115 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/hadoop.log.dir in system properties and HBase conf 2023-10-27 23:52:40,115 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/mapreduce.cluster.local.dir in system properties and HBase conf 2023-10-27 23:52:40,116 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-10-27 23:52:40,116 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-10-27 23:52:40,235 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-10-27 23:52:40,643 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-10-27 23:52:40,649 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-10-27 23:52:40,650 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-10-27 23:52:40,650 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-10-27 23:52:40,650 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-10-27 23:52:40,651 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-10-27 23:52:40,651 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-10-27 23:52:40,652 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-10-27 23:52:40,652 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/dfs.journalnode.edits.dir in system properties and HBase conf 2023-10-27 23:52:40,653 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-10-27 23:52:40,653 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/nfs.dump.dir in system properties and HBase conf 2023-10-27 23:52:40,654 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/java.io.tmpdir in system properties and HBase conf 2023-10-27 23:52:40,654 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/dfs.journalnode.edits.dir in system properties and HBase conf 2023-10-27 23:52:40,654 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-10-27 23:52:40,655 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-10-27 23:52:41,349 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-10-27 23:52:41,353 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-10-27 23:52:41,856 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-10-27 23:52:42,043 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-10-27 23:52:42,057 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-10-27 23:52:42,090 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-10-27 23:52:42,120 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/java.io.tmpdir/Jetty_localhost_44705_hdfs____.k9r0ii/webapp 2023-10-27 23:52:42,241 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44705 2023-10-27 23:52:42,250 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-10-27 23:52:42,250 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-10-27 23:52:43,476 WARN [Listener at localhost/46525] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-10-27 23:52:43,545 WARN [Listener at localhost/46525] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-10-27 23:52:43,563 WARN [Listener at localhost/46525] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-10-27 23:52:43,569 INFO [Listener at localhost/46525] log.Slf4jLog(67): jetty-6.1.26 2023-10-27 23:52:43,574 INFO [Listener at localhost/46525] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/java.io.tmpdir/Jetty_localhost_45017_datanode____.dfz38/webapp 2023-10-27 23:52:43,654 INFO [Listener at localhost/46525] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45017 2023-10-27 23:52:43,909 WARN [Listener at localhost/37393] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-10-27 23:52:43,929 WARN [Listener at localhost/37393] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-10-27 23:52:43,934 WARN [Listener at localhost/37393] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-10-27 23:52:43,936 INFO [Listener at localhost/37393] log.Slf4jLog(67): jetty-6.1.26 2023-10-27 23:52:43,946 INFO [Listener at localhost/37393] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/java.io.tmpdir/Jetty_localhost_42225_datanode____54t2qm/webapp 2023-10-27 23:52:44,033 INFO [Listener at localhost/37393] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42225 2023-10-27 23:52:44,058 WARN [Listener at localhost/42471] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-10-27 23:52:44,094 WARN [Listener at localhost/42471] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-10-27 23:52:44,099 WARN [Listener at localhost/42471] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-10-27 23:52:44,101 INFO [Listener at localhost/42471] log.Slf4jLog(67): jetty-6.1.26 2023-10-27 23:52:44,108 INFO [Listener at localhost/42471] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/java.io.tmpdir/Jetty_localhost_36737_datanode____.fgg0ap/webapp 2023-10-27 23:52:44,199 INFO [Listener at localhost/42471] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36737 2023-10-27 23:52:44,209 WARN [Listener at localhost/43373] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-10-27 23:52:45,118 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc269a56da6593ed7: Processing first storage report for DS-f69b18a9-729f-4b4e-b8d0-c9d44deb873f from datanode 9d32eec7-ccca-436f-a2e6-4184d95cd961 2023-10-27 23:52:45,120 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc269a56da6593ed7: from storage DS-f69b18a9-729f-4b4e-b8d0-c9d44deb873f node DatanodeRegistration(127.0.0.1:45899, datanodeUuid=9d32eec7-ccca-436f-a2e6-4184d95cd961, infoPort=33223, infoSecurePort=0, ipcPort=43373, storageInfo=lv=-57;cid=testClusterID;nsid=627202959;c=1698450761445), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-10-27 23:52:45,120 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf5bb4e3e03cbcfb9: Processing first storage report for DS-2c9db347-7433-46b6-b474-b03d8822beb5 from datanode 41efbad2-7a69-4304-ba15-0342fa6d22c3 2023-10-27 23:52:45,120 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf5bb4e3e03cbcfb9: from storage DS-2c9db347-7433-46b6-b474-b03d8822beb5 node DatanodeRegistration(127.0.0.1:34883, datanodeUuid=41efbad2-7a69-4304-ba15-0342fa6d22c3, infoPort=43885, infoSecurePort=0, ipcPort=42471, storageInfo=lv=-57;cid=testClusterID;nsid=627202959;c=1698450761445), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-10-27 23:52:45,120 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x92509ed886e053a1: Processing first storage report for DS-0fbfb6c9-a905-49b8-b039-f7170e563ff7 from datanode 36ff42f7-94a3-4a82-a7a9-0f471d56a50f 2023-10-27 23:52:45,120 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x92509ed886e053a1: from storage DS-0fbfb6c9-a905-49b8-b039-f7170e563ff7 node DatanodeRegistration(127.0.0.1:44001, datanodeUuid=36ff42f7-94a3-4a82-a7a9-0f471d56a50f, infoPort=35607, infoSecurePort=0, ipcPort=37393, storageInfo=lv=-57;cid=testClusterID;nsid=627202959;c=1698450761445), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-10-27 23:52:45,120 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc269a56da6593ed7: Processing first storage report for DS-f2443f23-e699-472f-915c-f9064e917e67 from datanode 9d32eec7-ccca-436f-a2e6-4184d95cd961 2023-10-27 23:52:45,120 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc269a56da6593ed7: from storage DS-f2443f23-e699-472f-915c-f9064e917e67 node DatanodeRegistration(127.0.0.1:45899, datanodeUuid=9d32eec7-ccca-436f-a2e6-4184d95cd961, infoPort=33223, infoSecurePort=0, ipcPort=43373, storageInfo=lv=-57;cid=testClusterID;nsid=627202959;c=1698450761445), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-10-27 23:52:45,121 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf5bb4e3e03cbcfb9: Processing first storage report for DS-9915a6e8-47dc-48fa-a2a3-1d8b18e6b279 from datanode 41efbad2-7a69-4304-ba15-0342fa6d22c3 2023-10-27 23:52:45,121 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf5bb4e3e03cbcfb9: from storage DS-9915a6e8-47dc-48fa-a2a3-1d8b18e6b279 node DatanodeRegistration(127.0.0.1:34883, datanodeUuid=41efbad2-7a69-4304-ba15-0342fa6d22c3, infoPort=43885, infoSecurePort=0, ipcPort=42471, storageInfo=lv=-57;cid=testClusterID;nsid=627202959;c=1698450761445), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-10-27 23:52:45,121 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x92509ed886e053a1: Processing first storage report for DS-b3cea8a6-1c1e-40e7-8ce1-c2b26bc162f6 from datanode 36ff42f7-94a3-4a82-a7a9-0f471d56a50f 2023-10-27 23:52:45,121 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x92509ed886e053a1: from storage DS-b3cea8a6-1c1e-40e7-8ce1-c2b26bc162f6 node DatanodeRegistration(127.0.0.1:44001, datanodeUuid=36ff42f7-94a3-4a82-a7a9-0f471d56a50f, infoPort=35607, infoSecurePort=0, ipcPort=37393, storageInfo=lv=-57;cid=testClusterID;nsid=627202959;c=1698450761445), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-10-27 23:52:45,161 DEBUG [Listener at localhost/43373] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21 2023-10-27 23:52:45,251 INFO [Listener at localhost/43373] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/zookeeper_0, clientPort=58988, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-10-27 23:52:45,267 INFO [Listener at localhost/43373] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58988 2023-10-27 23:52:45,277 INFO [Listener at localhost/43373] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-10-27 23:52:45,280 INFO [Listener at localhost/43373] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-10-27 23:52:45,918 INFO [Listener at localhost/43373] util.FSUtils(471): Created version file at hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5 with version=8 2023-10-27 23:52:45,919 INFO [Listener at localhost/43373] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/hbase-staging 2023-10-27 23:52:46,192 INFO [Listener at localhost/43373] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-27 23:52:46,577 INFO [Listener at localhost/43373] client.ConnectionUtils(127): master/jenkins-hbase16:0 server-side Connection retries=45 2023-10-27 23:52:46,603 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-10-27 23:52:46,604 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-10-27 23:52:46,604 INFO [Listener at localhost/43373] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-10-27 23:52:46,604 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-10-27 23:52:46,604 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-10-27 23:52:46,731 INFO [Listener at localhost/43373] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-10-27 23:52:46,791 DEBUG [Listener at localhost/43373] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-10-27 23:52:46,871 INFO [Listener at localhost/43373] ipc.NettyRpcServer(120): Bind to /188.40.62.62:40039 2023-10-27 23:52:46,879 INFO [Listener at localhost/43373] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-10-27 23:52:46,881 INFO [Listener at localhost/43373] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-10-27 23:52:46,901 INFO [Listener at localhost/43373] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40039 connecting to ZooKeeper ensemble=127.0.0.1:58988 2023-10-27 23:52:46,997 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:400390x0, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-10-27 23:52:47,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:40039-0x1018a4ce86c0000 connected 2023-10-27 23:52:47,072 DEBUG [Listener at localhost/43373] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-10-27 23:52:47,072 DEBUG [Listener at localhost/43373] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-10-27 23:52:47,076 DEBUG [Listener at localhost/43373] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-10-27 23:52:47,083 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40039 2023-10-27 23:52:47,083 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40039 2023-10-27 23:52:47,083 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40039 2023-10-27 23:52:47,084 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40039 2023-10-27 23:52:47,084 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40039 2023-10-27 23:52:47,089 INFO [Listener at localhost/43373] master.HMaster(444): hbase.rootdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5, hbase.cluster.distributed=false 2023-10-27 23:52:47,148 INFO [Listener at localhost/43373] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=45 2023-10-27 23:52:47,148 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-10-27 23:52:47,148 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-10-27 23:52:47,149 INFO [Listener at localhost/43373] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-10-27 23:52:47,149 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-10-27 23:52:47,149 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-10-27 23:52:47,153 INFO [Listener at localhost/43373] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-10-27 23:52:47,156 INFO [Listener at localhost/43373] ipc.NettyRpcServer(120): Bind to /188.40.62.62:39059 2023-10-27 23:52:47,158 INFO [Listener at localhost/43373] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-10-27 23:52:47,163 DEBUG [Listener at localhost/43373] mob.MobFileCache(121): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-10-27 23:52:47,165 INFO [Listener at localhost/43373] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-10-27 23:52:47,166 INFO [Listener at localhost/43373] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-10-27 23:52:47,168 INFO [Listener at localhost/43373] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39059 connecting to ZooKeeper ensemble=127.0.0.1:58988 2023-10-27 23:52:47,179 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:390590x0, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-10-27 23:52:47,180 DEBUG [Listener at localhost/43373] zookeeper.ZKUtil(164): regionserver:390590x0, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-10-27 23:52:47,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:39059-0x1018a4ce86c0001 connected 2023-10-27 23:52:47,181 DEBUG [Listener at localhost/43373] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-10-27 23:52:47,182 DEBUG [Listener at localhost/43373] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-10-27 23:52:47,183 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39059 2023-10-27 23:52:47,183 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39059 2023-10-27 23:52:47,184 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39059 2023-10-27 23:52:47,185 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39059 2023-10-27 23:52:47,185 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39059 2023-10-27 23:52:47,196 INFO [Listener at localhost/43373] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=45 2023-10-27 23:52:47,196 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-10-27 23:52:47,196 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-10-27 23:52:47,196 INFO [Listener at localhost/43373] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-10-27 23:52:47,197 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-10-27 23:52:47,197 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-10-27 23:52:47,197 INFO [Listener at localhost/43373] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-10-27 23:52:47,202 INFO [Listener at localhost/43373] ipc.NettyRpcServer(120): Bind to /188.40.62.62:43641 2023-10-27 23:52:47,203 INFO [Listener at localhost/43373] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-10-27 23:52:47,204 DEBUG [Listener at localhost/43373] mob.MobFileCache(121): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-10-27 23:52:47,205 INFO [Listener at localhost/43373] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-10-27 23:52:47,207 INFO [Listener at localhost/43373] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-10-27 23:52:47,210 INFO [Listener at localhost/43373] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43641 connecting to ZooKeeper ensemble=127.0.0.1:58988 2023-10-27 23:52:47,220 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:436410x0, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-10-27 23:52:47,222 DEBUG [Listener at localhost/43373] zookeeper.ZKUtil(164): regionserver:436410x0, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-10-27 23:52:47,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:43641-0x1018a4ce86c0002 connected 2023-10-27 23:52:47,222 DEBUG [Listener at localhost/43373] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-10-27 23:52:47,223 DEBUG [Listener at localhost/43373] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-10-27 23:52:47,227 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43641 2023-10-27 23:52:47,227 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43641 2023-10-27 23:52:47,227 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43641 2023-10-27 23:52:47,228 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43641 2023-10-27 23:52:47,231 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43641 2023-10-27 23:52:47,242 INFO [Listener at localhost/43373] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=45 2023-10-27 23:52:47,242 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-10-27 23:52:47,242 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-10-27 23:52:47,243 INFO [Listener at localhost/43373] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-10-27 23:52:47,243 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-10-27 23:52:47,243 INFO [Listener at localhost/43373] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-10-27 23:52:47,243 INFO [Listener at localhost/43373] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-10-27 23:52:47,244 INFO [Listener at localhost/43373] ipc.NettyRpcServer(120): Bind to /188.40.62.62:36599 2023-10-27 23:52:47,245 INFO [Listener at localhost/43373] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-10-27 23:52:47,246 DEBUG [Listener at localhost/43373] mob.MobFileCache(121): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-10-27 23:52:47,247 INFO [Listener at localhost/43373] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-10-27 23:52:47,248 INFO [Listener at localhost/43373] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-10-27 23:52:47,250 INFO [Listener at localhost/43373] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36599 connecting to ZooKeeper ensemble=127.0.0.1:58988 2023-10-27 23:52:47,262 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:365990x0, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-10-27 23:52:47,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:36599-0x1018a4ce86c0003 connected 2023-10-27 23:52:47,264 DEBUG [Listener at localhost/43373] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-10-27 23:52:47,265 DEBUG [Listener at localhost/43373] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-10-27 23:52:47,266 DEBUG [Listener at localhost/43373] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-10-27 23:52:47,266 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36599 2023-10-27 23:52:47,266 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36599 2023-10-27 23:52:47,267 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36599 2023-10-27 23:52:47,271 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36599 2023-10-27 23:52:47,271 DEBUG [Listener at localhost/43373] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36599 2023-10-27 23:52:47,273 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase16.apache.org,40039,1698450766041 2023-10-27 23:52:47,287 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-10-27 23:52:47,289 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase16.apache.org,40039,1698450766041 2023-10-27 23:52:47,312 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-10-27 23:52:47,312 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-10-27 23:52:47,312 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-10-27 23:52:47,312 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-10-27 23:52:47,312 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-10-27 23:52:47,314 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-10-27 23:52:47,315 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase16.apache.org,40039,1698450766041 from backup master directory 2023-10-27 23:52:47,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-10-27 23:52:47,325 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase16.apache.org,40039,1698450766041 2023-10-27 23:52:47,325 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-10-27 23:52:47,326 WARN [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-10-27 23:52:47,326 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase16.apache.org,40039,1698450766041 2023-10-27 23:52:47,328 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-10-27 23:52:47,330 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-10-27 23:52:47,417 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/hbase.id with ID: b480d3ba-2c69-479c-abe7-226bfeadf51a 2023-10-27 23:52:47,457 INFO [master/jenkins-hbase16:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-10-27 23:52:47,478 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-10-27 23:52:47,522 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0d742742 to 127.0.0.1:58988 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-10-27 23:52:47,561 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f6bcd92, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-10-27 23:52:47,584 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-10-27 23:52:47,586 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-10-27 23:52:47,599 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-10-27 23:52:47,600 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-10-27 23:52:47,602 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-10-27 23:52:47,606 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-10-27 23:52:47,608 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-10-27 23:52:47,638 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/data/master/store-tmp 2023-10-27 23:52:47,692 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:52:47,692 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-10-27 23:52:47,692 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-10-27 23:52:47,692 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-10-27 23:52:47,693 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-10-27 23:52:47,693 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-10-27 23:52:47,693 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-10-27 23:52:47,693 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-10-27 23:52:47,694 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/WALs/jenkins-hbase16.apache.org,40039,1698450766041 2023-10-27 23:52:47,713 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C40039%2C1698450766041, suffix=, logDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/WALs/jenkins-hbase16.apache.org,40039,1698450766041, archiveDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/oldWALs, maxLogs=10 2023-10-27 23:52:47,774 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34883,DS-2c9db347-7433-46b6-b474-b03d8822beb5,DISK] 2023-10-27 23:52:47,774 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45899,DS-f69b18a9-729f-4b4e-b8d0-c9d44deb873f,DISK] 2023-10-27 23:52:47,774 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44001,DS-0fbfb6c9-a905-49b8-b039-f7170e563ff7,DISK] 2023-10-27 23:52:47,782 DEBUG [RS-EventLoopGroup-5-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-10-27 23:52:47,848 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/WALs/jenkins-hbase16.apache.org,40039,1698450766041/jenkins-hbase16.apache.org%2C40039%2C1698450766041.1698450767722 2023-10-27 23:52:47,849 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45899,DS-f69b18a9-729f-4b4e-b8d0-c9d44deb873f,DISK], DatanodeInfoWithStorage[127.0.0.1:44001,DS-0fbfb6c9-a905-49b8-b039-f7170e563ff7,DISK], DatanodeInfoWithStorage[127.0.0.1:34883,DS-2c9db347-7433-46b6-b474-b03d8822beb5,DISK]] 2023-10-27 23:52:47,850 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:52:47,850 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:52:47,854 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-10-27 23:52:47,855 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-10-27 23:52:47,914 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-10-27 23:52:47,923 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-10-27 23:52:47,944 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-10-27 23:52:47,956 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:47,961 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-10-27 23:52:47,963 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-10-27 23:52:47,978 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-10-27 23:52:47,983 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-10-27 23:52:47,984 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63083470, jitterRate=-0.05998304486274719}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-10-27 23:52:47,984 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-10-27 23:52:47,985 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-10-27 23:52:48,003 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:52:48,003 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:52:48,006 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:52:48,008 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 1 msec 2023-10-27 23:52:48,036 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 28 msec 2023-10-27 23:52:48,037 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-10-27 23:52:48,059 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-10-27 23:52:48,065 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-10-27 23:52:48,097 INFO [master/jenkins-hbase16:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-10-27 23:52:48,101 INFO [master/jenkins-hbase16:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-10-27 23:52:48,103 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-10-27 23:52:48,109 INFO [master/jenkins-hbase16:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-10-27 23:52:48,114 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-10-27 23:52:48,150 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-10-27 23:52:48,152 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-10-27 23:52:48,152 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-10-27 23:52:48,168 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-10-27 23:52:48,178 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-10-27 23:52:48,178 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-10-27 23:52:48,178 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-10-27 23:52:48,178 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-10-27 23:52:48,179 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-10-27 23:52:48,179 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase16.apache.org,40039,1698450766041, sessionid=0x1018a4ce86c0000, setting cluster-up flag (Was=false) 2023-10-27 23:52:48,208 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-10-27 23:52:48,237 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-10-27 23:52:48,239 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase16.apache.org,40039,1698450766041 2023-10-27 23:52:48,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-10-27 23:52:48,287 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-10-27 23:52:48,289 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase16.apache.org,40039,1698450766041 2023-10-27 23:52:48,292 WARN [master/jenkins-hbase16:0:becomeActiveMaster] snapshot.SnapshotManager(304): Couldn't delete working snapshot directory: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.hbase-snapshot/.tmp 2023-10-27 23:52:48,375 INFO [RS:1;jenkins-hbase16:43641] regionserver.HRegionServer(951): ClusterId : b480d3ba-2c69-479c-abe7-226bfeadf51a 2023-10-27 23:52:48,375 INFO [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(951): ClusterId : b480d3ba-2c69-479c-abe7-226bfeadf51a 2023-10-27 23:52:48,375 INFO [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(951): ClusterId : b480d3ba-2c69-479c-abe7-226bfeadf51a 2023-10-27 23:52:48,380 DEBUG [RS:1;jenkins-hbase16:43641] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-10-27 23:52:48,380 DEBUG [RS:2;jenkins-hbase16:36599] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-10-27 23:52:48,380 DEBUG [RS:0;jenkins-hbase16:39059] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-10-27 23:52:48,392 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1028): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-10-27 23:52:48,401 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-10-27 23:52:48,401 DEBUG [RS:0;jenkins-hbase16:39059] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-10-27 23:52:48,401 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-10-27 23:52:48,401 DEBUG [RS:2;jenkins-hbase16:36599] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-10-27 23:52:48,402 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-10-27 23:52:48,401 DEBUG [RS:1;jenkins-hbase16:43641] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-10-27 23:52:48,402 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-10-27 23:52:48,402 DEBUG [RS:1;jenkins-hbase16:43641] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-10-27 23:52:48,402 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase16:0, corePoolSize=10, maxPoolSize=10 2023-10-27 23:52:48,402 DEBUG [RS:2;jenkins-hbase16:36599] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-10-27 23:52:48,402 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,401 DEBUG [RS:0;jenkins-hbase16:39059] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-10-27 23:52:48,402 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-10-27 23:52:48,403 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,403 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450798403 2023-10-27 23:52:48,406 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-10-27 23:52:48,411 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-10-27 23:52:48,411 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-10-27 23:52:48,413 DEBUG [RS:0;jenkins-hbase16:39059] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-10-27 23:52:48,413 DEBUG [RS:1;jenkins-hbase16:43641] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-10-27 23:52:48,413 DEBUG [RS:2;jenkins-hbase16:36599] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-10-27 23:52:48,417 DEBUG [RS:1;jenkins-hbase16:43641] zookeeper.ReadOnlyZKClient(139): Connect 0x4c00dde6 to 127.0.0.1:58988 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-10-27 23:52:48,418 DEBUG [RS:0;jenkins-hbase16:39059] zookeeper.ReadOnlyZKClient(139): Connect 0x023f0099 to 127.0.0.1:58988 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-10-27 23:52:48,418 DEBUG [RS:2;jenkins-hbase16:36599] zookeeper.ReadOnlyZKClient(139): Connect 0x4980ca39 to 127.0.0.1:58988 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-10-27 23:52:48,420 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-10-27 23:52:48,422 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-10-27 23:52:48,430 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-10-27 23:52:48,430 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-10-27 23:52:48,431 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-10-27 23:52:48,431 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-10-27 23:52:48,431 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,434 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-10-27 23:52:48,435 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-10-27 23:52:48,436 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-10-27 23:52:48,438 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-10-27 23:52:48,438 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-10-27 23:52:48,440 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1698450768440,5,FailOnTimeoutGroup] 2023-10-27 23:52:48,441 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1698450768440,5,FailOnTimeoutGroup] 2023-10-27 23:52:48,441 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,441 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-10-27 23:52:48,443 DEBUG [RS:1;jenkins-hbase16:43641] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42c73a16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-10-27 23:52:48,444 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,443 DEBUG [RS:0;jenkins-hbase16:39059] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@cf8feae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-10-27 23:52:48,444 DEBUG [RS:1;jenkins-hbase16:43641] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46f8b696, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-10-27 23:52:48,444 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,444 DEBUG [RS:2;jenkins-hbase16:36599] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47473d79, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-10-27 23:52:48,444 DEBUG [RS:0;jenkins-hbase16:39059] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8c76be9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-10-27 23:52:48,445 DEBUG [RS:2;jenkins-hbase16:36599] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4378a149, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-10-27 23:52:48,474 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-10-27 23:52:48,475 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-10-27 23:52:48,475 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5 2023-10-27 23:52:48,480 DEBUG [RS:0;jenkins-hbase16:39059] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase16:39059 2023-10-27 23:52:48,481 DEBUG [RS:1;jenkins-hbase16:43641] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase16:43641 2023-10-27 23:52:48,482 DEBUG [RS:2;jenkins-hbase16:36599] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase16:36599 2023-10-27 23:52:48,486 INFO [RS:0;jenkins-hbase16:39059] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-10-27 23:52:48,486 INFO [RS:1;jenkins-hbase16:43641] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-10-27 23:52:48,487 INFO [RS:1;jenkins-hbase16:43641] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-10-27 23:52:48,486 INFO [RS:2;jenkins-hbase16:36599] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-10-27 23:52:48,487 DEBUG [RS:1;jenkins-hbase16:43641] regionserver.HRegionServer(1022): About to register with Master. 2023-10-27 23:52:48,487 INFO [RS:2;jenkins-hbase16:36599] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-10-27 23:52:48,487 INFO [RS:0;jenkins-hbase16:39059] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-10-27 23:52:48,487 DEBUG [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(1022): About to register with Master. 2023-10-27 23:52:48,487 DEBUG [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(1022): About to register with Master. 2023-10-27 23:52:48,493 INFO [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase16.apache.org,40039,1698450766041 with isa=jenkins-hbase16.apache.org/188.40.62.62:39059, startcode=1698450767147 2023-10-27 23:52:48,492 INFO [RS:1;jenkins-hbase16:43641] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase16.apache.org,40039,1698450766041 with isa=jenkins-hbase16.apache.org/188.40.62.62:43641, startcode=1698450767195 2023-10-27 23:52:48,493 INFO [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase16.apache.org,40039,1698450766041 with isa=jenkins-hbase16.apache.org/188.40.62.62:36599, startcode=1698450767241 2023-10-27 23:52:48,505 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:52:48,509 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-10-27 23:52:48,513 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/info 2023-10-27 23:52:48,514 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-10-27 23:52:48,517 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:48,517 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-10-27 23:52:48,520 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/rep_barrier 2023-10-27 23:52:48,520 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-10-27 23:52:48,521 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:48,522 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-10-27 23:52:48,522 DEBUG [RS:2;jenkins-hbase16:36599] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-10-27 23:52:48,522 DEBUG [RS:0;jenkins-hbase16:39059] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-10-27 23:52:48,522 DEBUG [RS:1;jenkins-hbase16:43641] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-10-27 23:52:48,525 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/table 2023-10-27 23:52:48,526 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-10-27 23:52:48,527 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:48,530 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740 2023-10-27 23:52:48,531 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740 2023-10-27 23:52:48,534 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-10-27 23:52:48,536 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-10-27 23:52:48,541 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-10-27 23:52:48,543 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59245303, jitterRate=-0.11717619001865387}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-10-27 23:52:48,543 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-10-27 23:52:48,543 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-10-27 23:52:48,544 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-10-27 23:52:48,544 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-10-27 23:52:48,544 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-10-27 23:52:48,544 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-10-27 23:52:48,551 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-10-27 23:52:48,552 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-10-27 23:52:48,568 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-10-27 23:52:48,568 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-10-27 23:52:48,576 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-10-27 23:52:48,587 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:43711, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-10-27 23:52:48,587 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:53281, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-10-27 23:52:48,587 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:43259, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-10-27 23:52:48,594 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-10-27 23:52:48,597 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-10-27 23:52:48,600 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40039] master.ServerManager(388): Registering regionserver=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:48,609 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40039] master.ServerManager(388): Registering regionserver=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:48,610 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40039] master.ServerManager(388): Registering regionserver=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:48,615 DEBUG [RS:1;jenkins-hbase16:43641] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5 2023-10-27 23:52:48,615 DEBUG [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5 2023-10-27 23:52:48,615 DEBUG [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5 2023-10-27 23:52:48,615 DEBUG [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:46525 2023-10-27 23:52:48,615 DEBUG [RS:1;jenkins-hbase16:43641] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:46525 2023-10-27 23:52:48,616 DEBUG [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-10-27 23:52:48,615 DEBUG [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:46525 2023-10-27 23:52:48,616 DEBUG [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-10-27 23:52:48,616 DEBUG [RS:1;jenkins-hbase16:43641] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-10-27 23:52:48,651 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-10-27 23:52:48,662 DEBUG [RS:0;jenkins-hbase16:39059] zookeeper.ZKUtil(162): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:48,662 DEBUG [RS:2;jenkins-hbase16:36599] zookeeper.ZKUtil(162): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:48,662 WARN [RS:0;jenkins-hbase16:39059] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-10-27 23:52:48,662 WARN [RS:2;jenkins-hbase16:36599] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-10-27 23:52:48,663 INFO [RS:0;jenkins-hbase16:39059] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-10-27 23:52:48,662 DEBUG [RS:1;jenkins-hbase16:43641] zookeeper.ZKUtil(162): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:48,663 INFO [RS:2;jenkins-hbase16:36599] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-10-27 23:52:48,663 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,43641,1698450767195] 2023-10-27 23:52:48,663 DEBUG [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(1948): logDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/WALs/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:48,663 WARN [RS:1;jenkins-hbase16:43641] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-10-27 23:52:48,664 DEBUG [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(1948): logDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/WALs/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:48,664 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,36599,1698450767241] 2023-10-27 23:52:48,664 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,39059,1698450767147] 2023-10-27 23:52:48,664 INFO [RS:1;jenkins-hbase16:43641] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-10-27 23:52:48,664 DEBUG [RS:1;jenkins-hbase16:43641] regionserver.HRegionServer(1948): logDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/WALs/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:48,676 DEBUG [RS:0;jenkins-hbase16:39059] zookeeper.ZKUtil(162): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:48,676 DEBUG [RS:2;jenkins-hbase16:36599] zookeeper.ZKUtil(162): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:48,676 DEBUG [RS:1;jenkins-hbase16:43641] zookeeper.ZKUtil(162): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:48,676 DEBUG [RS:0;jenkins-hbase16:39059] zookeeper.ZKUtil(162): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:48,677 DEBUG [RS:2;jenkins-hbase16:36599] zookeeper.ZKUtil(162): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:48,677 DEBUG [RS:1;jenkins-hbase16:43641] zookeeper.ZKUtil(162): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:48,677 DEBUG [RS:2;jenkins-hbase16:36599] zookeeper.ZKUtil(162): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:48,677 DEBUG [RS:0;jenkins-hbase16:39059] zookeeper.ZKUtil(162): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:48,677 DEBUG [RS:1;jenkins-hbase16:43641] zookeeper.ZKUtil(162): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:48,687 DEBUG [RS:0;jenkins-hbase16:39059] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-10-27 23:52:48,687 DEBUG [RS:2;jenkins-hbase16:36599] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-10-27 23:52:48,687 DEBUG [RS:1;jenkins-hbase16:43641] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-10-27 23:52:48,696 INFO [RS:2;jenkins-hbase16:36599] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-10-27 23:52:48,696 INFO [RS:1;jenkins-hbase16:43641] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-10-27 23:52:48,696 INFO [RS:0;jenkins-hbase16:39059] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-10-27 23:52:48,718 INFO [RS:2;jenkins-hbase16:36599] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-10-27 23:52:48,718 INFO [RS:0;jenkins-hbase16:39059] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-10-27 23:52:48,718 INFO [RS:1;jenkins-hbase16:43641] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-10-27 23:52:48,722 INFO [RS:0;jenkins-hbase16:39059] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-10-27 23:52:48,722 INFO [RS:1;jenkins-hbase16:43641] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-10-27 23:52:48,723 INFO [RS:0;jenkins-hbase16:39059] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,723 INFO [RS:1;jenkins-hbase16:43641] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,722 INFO [RS:2;jenkins-hbase16:36599] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-10-27 23:52:48,723 INFO [RS:2;jenkins-hbase16:36599] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,724 INFO [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-10-27 23:52:48,724 INFO [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-10-27 23:52:48,724 INFO [RS:1;jenkins-hbase16:43641] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-10-27 23:52:48,732 INFO [RS:0;jenkins-hbase16:39059] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,732 INFO [RS:1;jenkins-hbase16:43641] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,732 INFO [RS:2;jenkins-hbase16:36599] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,732 DEBUG [RS:0;jenkins-hbase16:39059] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,732 DEBUG [RS:1;jenkins-hbase16:43641] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,732 DEBUG [RS:0;jenkins-hbase16:39059] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,732 DEBUG [RS:1;jenkins-hbase16:43641] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,732 DEBUG [RS:2;jenkins-hbase16:36599] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,733 DEBUG [RS:1;jenkins-hbase16:43641] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,733 DEBUG [RS:2;jenkins-hbase16:36599] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,733 DEBUG [RS:1;jenkins-hbase16:43641] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,733 DEBUG [RS:2;jenkins-hbase16:36599] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,733 DEBUG [RS:1;jenkins-hbase16:43641] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,733 DEBUG [RS:2;jenkins-hbase16:36599] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,733 DEBUG [RS:1;jenkins-hbase16:43641] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-10-27 23:52:48,733 DEBUG [RS:2;jenkins-hbase16:36599] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,733 DEBUG [RS:1;jenkins-hbase16:43641] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,733 DEBUG [RS:2;jenkins-hbase16:36599] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-10-27 23:52:48,733 DEBUG [RS:1;jenkins-hbase16:43641] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,733 DEBUG [RS:2;jenkins-hbase16:36599] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,733 DEBUG [RS:1;jenkins-hbase16:43641] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,733 DEBUG [RS:2;jenkins-hbase16:36599] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,734 DEBUG [RS:1;jenkins-hbase16:43641] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,734 DEBUG [RS:2;jenkins-hbase16:36599] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,733 DEBUG [RS:0;jenkins-hbase16:39059] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,734 DEBUG [RS:2;jenkins-hbase16:36599] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,734 DEBUG [RS:0;jenkins-hbase16:39059] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,735 DEBUG [RS:0;jenkins-hbase16:39059] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,735 DEBUG [RS:0;jenkins-hbase16:39059] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-10-27 23:52:48,735 DEBUG [RS:0;jenkins-hbase16:39059] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,735 DEBUG [RS:0;jenkins-hbase16:39059] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,735 DEBUG [RS:0;jenkins-hbase16:39059] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,736 DEBUG [RS:0;jenkins-hbase16:39059] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-10-27 23:52:48,736 INFO [RS:1;jenkins-hbase16:43641] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,736 INFO [RS:1;jenkins-hbase16:43641] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,736 INFO [RS:2;jenkins-hbase16:36599] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,736 INFO [RS:1;jenkins-hbase16:43641] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,736 INFO [RS:2;jenkins-hbase16:36599] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,737 INFO [RS:2;jenkins-hbase16:36599] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,742 INFO [RS:0;jenkins-hbase16:39059] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,742 INFO [RS:0;jenkins-hbase16:39059] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,742 INFO [RS:0;jenkins-hbase16:39059] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,749 DEBUG [jenkins-hbase16:40039] assignment.AssignmentManager(2186): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-10-27 23:52:48,753 DEBUG [jenkins-hbase16:40039] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-10-27 23:52:48,758 INFO [RS:1;jenkins-hbase16:43641] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-10-27 23:52:48,763 INFO [RS:2;jenkins-hbase16:36599] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-10-27 23:52:48,758 INFO [RS:0;jenkins-hbase16:39059] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-10-27 23:52:48,765 DEBUG [jenkins-hbase16:40039] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-27 23:52:48,765 DEBUG [jenkins-hbase16:40039] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-10-27 23:52:48,765 DEBUG [jenkins-hbase16:40039] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-10-27 23:52:48,765 DEBUG [jenkins-hbase16:40039] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-10-27 23:52:48,768 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,36599,1698450767241, state=OPENING 2023-10-27 23:52:48,768 INFO [RS:2;jenkins-hbase16:36599] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,36599,1698450767241-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,768 INFO [RS:0;jenkins-hbase16:39059] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,39059,1698450767147-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,768 INFO [RS:1;jenkins-hbase16:43641] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,43641,1698450767195-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:48,783 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-10-27 23:52:48,791 INFO [RS:2;jenkins-hbase16:36599] regionserver.Replication(203): jenkins-hbase16.apache.org,36599,1698450767241 started 2023-10-27 23:52:48,791 INFO [RS:0;jenkins-hbase16:39059] regionserver.Replication(203): jenkins-hbase16.apache.org,39059,1698450767147 started 2023-10-27 23:52:48,791 INFO [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(1637): Serving as jenkins-hbase16.apache.org,36599,1698450767241, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:36599, sessionid=0x1018a4ce86c0003 2023-10-27 23:52:48,791 INFO [RS:1;jenkins-hbase16:43641] regionserver.Replication(203): jenkins-hbase16.apache.org,43641,1698450767195 started 2023-10-27 23:52:48,791 INFO [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(1637): Serving as jenkins-hbase16.apache.org,39059,1698450767147, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:39059, sessionid=0x1018a4ce86c0001 2023-10-27 23:52:48,791 INFO [RS:1;jenkins-hbase16:43641] regionserver.HRegionServer(1637): Serving as jenkins-hbase16.apache.org,43641,1698450767195, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:43641, sessionid=0x1018a4ce86c0002 2023-10-27 23:52:48,791 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-10-27 23:52:48,791 DEBUG [RS:0;jenkins-hbase16:39059] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-10-27 23:52:48,791 DEBUG [RS:1;jenkins-hbase16:43641] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-10-27 23:52:48,791 DEBUG [RS:2;jenkins-hbase16:36599] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-10-27 23:52:48,791 DEBUG [RS:0;jenkins-hbase16:39059] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:48,792 DEBUG [RS:2;jenkins-hbase16:36599] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:48,792 DEBUG [RS:0;jenkins-hbase16:39059] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,39059,1698450767147' 2023-10-27 23:52:48,792 DEBUG [RS:2;jenkins-hbase16:36599] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,36599,1698450767241' 2023-10-27 23:52:48,794 DEBUG [RS:2;jenkins-hbase16:36599] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:48,791 DEBUG [RS:1;jenkins-hbase16:43641] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:48,794 DEBUG [RS:0;jenkins-hbase16:39059] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:48,794 DEBUG [RS:1;jenkins-hbase16:43641] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,43641,1698450767195' 2023-10-27 23:52:48,794 DEBUG [RS:1;jenkins-hbase16:43641] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:48,794 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-10-27 23:52:48,795 DEBUG [RS:2;jenkins-hbase16:36599] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:48,796 DEBUG [RS:2;jenkins-hbase16:36599] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-10-27 23:52:48,796 DEBUG [RS:2;jenkins-hbase16:36599] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-10-27 23:52:48,796 DEBUG [RS:2;jenkins-hbase16:36599] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:48,796 DEBUG [RS:2;jenkins-hbase16:36599] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,36599,1698450767241' 2023-10-27 23:52:48,796 DEBUG [RS:2;jenkins-hbase16:36599] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-10-27 23:52:48,795 DEBUG [RS:1;jenkins-hbase16:43641] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:48,797 DEBUG [RS:2;jenkins-hbase16:36599] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-10-27 23:52:48,797 DEBUG [RS:1;jenkins-hbase16:43641] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-10-27 23:52:48,798 DEBUG [RS:1;jenkins-hbase16:43641] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-10-27 23:52:48,798 DEBUG [RS:1;jenkins-hbase16:43641] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:48,795 DEBUG [RS:0;jenkins-hbase16:39059] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:48,798 DEBUG [RS:1;jenkins-hbase16:43641] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,43641,1698450767195' 2023-10-27 23:52:48,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:52:48,798 DEBUG [RS:1;jenkins-hbase16:43641] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-10-27 23:52:48,798 DEBUG [RS:2;jenkins-hbase16:36599] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-10-27 23:52:48,799 INFO [RS:2;jenkins-hbase16:36599] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-10-27 23:52:48,799 DEBUG [RS:0;jenkins-hbase16:39059] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-10-27 23:52:48,799 DEBUG [RS:1;jenkins-hbase16:43641] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-10-27 23:52:48,799 INFO [RS:2;jenkins-hbase16:36599] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-10-27 23:52:48,799 DEBUG [RS:0;jenkins-hbase16:39059] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-10-27 23:52:48,799 DEBUG [RS:0;jenkins-hbase16:39059] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:48,799 DEBUG [RS:0;jenkins-hbase16:39059] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,39059,1698450767147' 2023-10-27 23:52:48,799 DEBUG [RS:0;jenkins-hbase16:39059] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-10-27 23:52:48,799 DEBUG [RS:1;jenkins-hbase16:43641] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-10-27 23:52:48,800 INFO [RS:1;jenkins-hbase16:43641] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-10-27 23:52:48,800 INFO [RS:1;jenkins-hbase16:43641] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-10-27 23:52:48,800 DEBUG [RS:0;jenkins-hbase16:39059] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-10-27 23:52:48,800 DEBUG [RS:0;jenkins-hbase16:39059] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-10-27 23:52:48,800 INFO [RS:0;jenkins-hbase16:39059] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-10-27 23:52:48,800 INFO [RS:0;jenkins-hbase16:39059] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-10-27 23:52:48,909 INFO [RS:1;jenkins-hbase16:43641] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C43641%2C1698450767195, suffix=, logDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/WALs/jenkins-hbase16.apache.org,43641,1698450767195, archiveDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/oldWALs, maxLogs=32 2023-10-27 23:52:48,909 INFO [RS:0;jenkins-hbase16:39059] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C39059%2C1698450767147, suffix=, logDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/WALs/jenkins-hbase16.apache.org,39059,1698450767147, archiveDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/oldWALs, maxLogs=32 2023-10-27 23:52:48,909 INFO [RS:2;jenkins-hbase16:36599] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C36599%2C1698450767241, suffix=, logDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/WALs/jenkins-hbase16.apache.org,36599,1698450767241, archiveDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/oldWALs, maxLogs=32 2023-10-27 23:52:48,928 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44001,DS-0fbfb6c9-a905-49b8-b039-f7170e563ff7,DISK] 2023-10-27 23:52:48,928 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45899,DS-f69b18a9-729f-4b4e-b8d0-c9d44deb873f,DISK] 2023-10-27 23:52:48,928 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34883,DS-2c9db347-7433-46b6-b474-b03d8822beb5,DISK] 2023-10-27 23:52:48,935 INFO [RS:0;jenkins-hbase16:39059] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/WALs/jenkins-hbase16.apache.org,39059,1698450767147/jenkins-hbase16.apache.org%2C39059%2C1698450767147.1698450768912 2023-10-27 23:52:48,940 DEBUG [RS:0;jenkins-hbase16:39059] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44001,DS-0fbfb6c9-a905-49b8-b039-f7170e563ff7,DISK], DatanodeInfoWithStorage[127.0.0.1:34883,DS-2c9db347-7433-46b6-b474-b03d8822beb5,DISK], DatanodeInfoWithStorage[127.0.0.1:45899,DS-f69b18a9-729f-4b4e-b8d0-c9d44deb873f,DISK]] 2023-10-27 23:52:48,944 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45899,DS-f69b18a9-729f-4b4e-b8d0-c9d44deb873f,DISK] 2023-10-27 23:52:48,945 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44001,DS-0fbfb6c9-a905-49b8-b039-f7170e563ff7,DISK] 2023-10-27 23:52:48,946 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44001,DS-0fbfb6c9-a905-49b8-b039-f7170e563ff7,DISK] 2023-10-27 23:52:48,946 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34883,DS-2c9db347-7433-46b6-b474-b03d8822beb5,DISK] 2023-10-27 23:52:48,946 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34883,DS-2c9db347-7433-46b6-b474-b03d8822beb5,DISK] 2023-10-27 23:52:48,947 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45899,DS-f69b18a9-729f-4b4e-b8d0-c9d44deb873f,DISK] 2023-10-27 23:52:48,963 INFO [RS:1;jenkins-hbase16:43641] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/WALs/jenkins-hbase16.apache.org,43641,1698450767195/jenkins-hbase16.apache.org%2C43641%2C1698450767195.1698450768913 2023-10-27 23:52:48,967 DEBUG [RS:1;jenkins-hbase16:43641] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45899,DS-f69b18a9-729f-4b4e-b8d0-c9d44deb873f,DISK], DatanodeInfoWithStorage[127.0.0.1:44001,DS-0fbfb6c9-a905-49b8-b039-f7170e563ff7,DISK], DatanodeInfoWithStorage[127.0.0.1:34883,DS-2c9db347-7433-46b6-b474-b03d8822beb5,DISK]] 2023-10-27 23:52:48,976 INFO [RS:2;jenkins-hbase16:36599] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/WALs/jenkins-hbase16.apache.org,36599,1698450767241/jenkins-hbase16.apache.org%2C36599%2C1698450767241.1698450768913 2023-10-27 23:52:48,976 DEBUG [RS:2;jenkins-hbase16:36599] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34883,DS-2c9db347-7433-46b6-b474-b03d8822beb5,DISK], DatanodeInfoWithStorage[127.0.0.1:44001,DS-0fbfb6c9-a905-49b8-b039-f7170e563ff7,DISK], DatanodeInfoWithStorage[127.0.0.1:45899,DS-f69b18a9-729f-4b4e-b8d0-c9d44deb873f,DISK]] 2023-10-27 23:52:48,995 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:48,998 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-10-27 23:52:49,000 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:49772, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-10-27 23:52:49,014 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-10-27 23:52:49,015 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-10-27 23:52:49,020 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C36599%2C1698450767241.meta, suffix=.meta, logDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/WALs/jenkins-hbase16.apache.org,36599,1698450767241, archiveDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/oldWALs, maxLogs=32 2023-10-27 23:52:49,036 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44001,DS-0fbfb6c9-a905-49b8-b039-f7170e563ff7,DISK] 2023-10-27 23:52:49,038 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34883,DS-2c9db347-7433-46b6-b474-b03d8822beb5,DISK] 2023-10-27 23:52:49,039 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45899,DS-f69b18a9-729f-4b4e-b8d0-c9d44deb873f,DISK] 2023-10-27 23:52:49,046 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/WALs/jenkins-hbase16.apache.org,36599,1698450767241/jenkins-hbase16.apache.org%2C36599%2C1698450767241.meta.1698450769021.meta 2023-10-27 23:52:49,047 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44001,DS-0fbfb6c9-a905-49b8-b039-f7170e563ff7,DISK], DatanodeInfoWithStorage[127.0.0.1:34883,DS-2c9db347-7433-46b6-b474-b03d8822beb5,DISK], DatanodeInfoWithStorage[127.0.0.1:45899,DS-f69b18a9-729f-4b4e-b8d0-c9d44deb873f,DISK]] 2023-10-27 23:52:49,047 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:52:49,051 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:52:49,052 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-10-27 23:52:49,066 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-10-27 23:52:49,068 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-10-27 23:52:49,072 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-10-27 23:52:49,072 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:52:49,072 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-10-27 23:52:49,072 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-10-27 23:52:49,075 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-10-27 23:52:49,078 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/info 2023-10-27 23:52:49,078 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/info 2023-10-27 23:52:49,078 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-10-27 23:52:49,080 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:49,080 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-10-27 23:52:49,081 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/rep_barrier 2023-10-27 23:52:49,082 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/rep_barrier 2023-10-27 23:52:49,083 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-10-27 23:52:49,085 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:49,086 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-10-27 23:52:49,088 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/table 2023-10-27 23:52:49,089 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/table 2023-10-27 23:52:49,089 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-10-27 23:52:49,090 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:49,092 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740 2023-10-27 23:52:49,095 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740 2023-10-27 23:52:49,098 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-10-27 23:52:49,100 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-10-27 23:52:49,101 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66926770, jitterRate=-0.002713412046432495}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-10-27 23:52:49,101 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-10-27 23:52:49,110 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1698450768986 2023-10-27 23:52:49,126 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-10-27 23:52:49,127 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-10-27 23:52:49,127 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,36599,1698450767241, state=OPEN 2023-10-27 23:52:49,136 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-10-27 23:52:49,137 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-10-27 23:52:49,141 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=3, resume processing ppid=2 2023-10-27 23:52:49,142 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,36599,1698450767241 in 339 msec 2023-10-27 23:52:49,148 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=2, resume processing ppid=1 2023-10-27 23:52:49,148 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 567 msec 2023-10-27 23:52:49,155 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 820 msec 2023-10-27 23:52:49,155 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1698450769155, completionTime=-1 2023-10-27 23:52:49,155 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ServerManager(808): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-10-27 23:52:49,155 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:52:49,217 DEBUG [hconnection-0x27f582be-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-10-27 23:52:49,220 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:49780, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-10-27 23:52:49,235 INFO [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:52:49,235 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450829235 2023-10-27 23:52:49,235 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450889235 2023-10-27 23:52:49,235 INFO [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1546): Joined the cluster in 79 msec 2023-10-27 23:52:49,295 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,40039,1698450766041-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:49,295 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,40039,1698450766041-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:49,295 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,40039,1698450766041-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:49,296 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase16:40039, period=300000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:49,297 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-10-27 23:52:49,304 DEBUG [master/jenkins-hbase16:0.Chore.1] janitor.CatalogJanitor(175): 2023-10-27 23:52:49,314 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-10-27 23:52:49,317 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-10-27 23:52:49,325 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1028): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-10-27 23:52:49,328 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-10-27 23:52:49,331 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-10-27 23:52:49,354 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/hbase/namespace/5005e44f9940b2c7d862a285ee6d8687 2023-10-27 23:52:49,356 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/hbase/namespace/5005e44f9940b2c7d862a285ee6d8687 empty. 2023-10-27 23:52:49,357 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/hbase/namespace/5005e44f9940b2c7d862a285ee6d8687 2023-10-27 23:52:49,357 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-10-27 23:52:49,399 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-10-27 23:52:49,401 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5005e44f9940b2c7d862a285ee6d8687, NAME => 'hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp 2023-10-27 23:52:49,418 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:52:49,418 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 5005e44f9940b2c7d862a285ee6d8687, disabling compactions & flushes 2023-10-27 23:52:49,419 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. 2023-10-27 23:52:49,419 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. 2023-10-27 23:52:49,419 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. after waiting 0 ms 2023-10-27 23:52:49,419 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. 2023-10-27 23:52:49,419 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. 2023-10-27 23:52:49,420 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 5005e44f9940b2c7d862a285ee6d8687: 2023-10-27 23:52:49,425 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-10-27 23:52:49,439 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1698450769428"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450769428"}]},"ts":"1698450769428"} 2023-10-27 23:52:49,466 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-10-27 23:52:49,468 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-10-27 23:52:49,472 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450769468"}]},"ts":"1698450769468"} 2023-10-27 23:52:49,476 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-10-27 23:52:49,496 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-10-27 23:52:49,499 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-27 23:52:49,499 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-10-27 23:52:49,499 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-10-27 23:52:49,500 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-10-27 23:52:49,503 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=5005e44f9940b2c7d862a285ee6d8687, ASSIGN}] 2023-10-27 23:52:49,508 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=5005e44f9940b2c7d862a285ee6d8687, ASSIGN 2023-10-27 23:52:49,509 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=5005e44f9940b2c7d862a285ee6d8687, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,39059,1698450767147; forceNewPlan=false, retain=false 2023-10-27 23:52:49,661 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-10-27 23:52:49,662 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=5005e44f9940b2c7d862a285ee6d8687, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:49,663 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1698450769662"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450769662"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450769662"}]},"ts":"1698450769662"} 2023-10-27 23:52:49,669 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 5005e44f9940b2c7d862a285ee6d8687, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:52:49,824 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:49,824 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-10-27 23:52:49,827 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:50942, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-10-27 23:52:49,835 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. 2023-10-27 23:52:49,837 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5005e44f9940b2c7d862a285ee6d8687, NAME => 'hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:52:49,840 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:52:49,840 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 5005e44f9940b2c7d862a285ee6d8687 2023-10-27 23:52:49,841 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:52:49,841 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 5005e44f9940b2c7d862a285ee6d8687 2023-10-27 23:52:49,841 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 5005e44f9940b2c7d862a285ee6d8687 2023-10-27 23:52:49,844 INFO [StoreOpener-5005e44f9940b2c7d862a285ee6d8687-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 5005e44f9940b2c7d862a285ee6d8687 2023-10-27 23:52:49,846 DEBUG [StoreOpener-5005e44f9940b2c7d862a285ee6d8687-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/namespace/5005e44f9940b2c7d862a285ee6d8687/info 2023-10-27 23:52:49,846 DEBUG [StoreOpener-5005e44f9940b2c7d862a285ee6d8687-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/namespace/5005e44f9940b2c7d862a285ee6d8687/info 2023-10-27 23:52:49,847 INFO [StoreOpener-5005e44f9940b2c7d862a285ee6d8687-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5005e44f9940b2c7d862a285ee6d8687 columnFamilyName info 2023-10-27 23:52:49,848 INFO [StoreOpener-5005e44f9940b2c7d862a285ee6d8687-1] regionserver.HStore(310): Store=5005e44f9940b2c7d862a285ee6d8687/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:49,850 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/namespace/5005e44f9940b2c7d862a285ee6d8687 2023-10-27 23:52:49,852 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/namespace/5005e44f9940b2c7d862a285ee6d8687 2023-10-27 23:52:49,856 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 5005e44f9940b2c7d862a285ee6d8687 2023-10-27 23:52:49,861 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/namespace/5005e44f9940b2c7d862a285ee6d8687/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-10-27 23:52:49,862 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 5005e44f9940b2c7d862a285ee6d8687; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73241300, jitterRate=0.09138041734695435}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-10-27 23:52:49,862 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 5005e44f9940b2c7d862a285ee6d8687: 2023-10-27 23:52:49,864 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687., pid=6, masterSystemTime=1698450769823 2023-10-27 23:52:49,869 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. 2023-10-27 23:52:49,870 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. 2023-10-27 23:52:49,871 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:49,872 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1698450769871"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450769871"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450769871"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450769871"}]},"ts":"1698450769871"} 2023-10-27 23:52:49,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=6, resume processing ppid=5 2023-10-27 23:52:49,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 5005e44f9940b2c7d862a285ee6d8687, server=jenkins-hbase16.apache.org,39059,1698450767147 in 206 msec 2023-10-27 23:52:49,883 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=5, resume processing ppid=4 2023-10-27 23:52:49,883 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=5005e44f9940b2c7d862a285ee6d8687, ASSIGN in 376 msec 2023-10-27 23:52:49,885 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-10-27 23:52:49,885 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450769885"}]},"ts":"1698450769885"} 2023-10-27 23:52:49,888 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-10-27 23:52:49,927 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-10-27 23:52:49,928 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-10-27 23:52:49,931 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 609 msec 2023-10-27 23:52:49,936 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-10-27 23:52:49,936 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-10-27 23:52:49,956 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-10-27 23:52:49,959 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:50952, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-10-27 23:52:49,971 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1028): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-10-27 23:52:50,000 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-10-27 23:52:50,013 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 47 msec 2023-10-27 23:52:50,025 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1028): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:52:50,045 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-10-27 23:52:50,058 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 31 msec 2023-10-27 23:52:50,086 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-10-27 23:52:50,103 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-10-27 23:52:50,103 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.777sec 2023-10-27 23:52:50,106 INFO [master/jenkins-hbase16:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-10-27 23:52:50,107 INFO [master/jenkins-hbase16:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-10-27 23:52:50,107 INFO [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-10-27 23:52:50,109 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,40039,1698450766041-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-10-27 23:52:50,109 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,40039,1698450766041-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-10-27 23:52:50,118 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-10-27 23:52:50,185 DEBUG [Listener at localhost/43373] zookeeper.ReadOnlyZKClient(139): Connect 0x28683c56 to 127.0.0.1:58988 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-10-27 23:52:50,196 DEBUG [Listener at localhost/43373] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c602b76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-10-27 23:52:50,209 DEBUG [hconnection-0x23f62efb-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-10-27 23:52:50,221 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:49794, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-10-27 23:52:50,230 INFO [Listener at localhost/43373] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase16.apache.org,40039,1698450766041 2023-10-27 23:52:50,260 INFO [Listener at localhost/43373] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionUnevenDaughter Thread=361, OpenFileDescriptor=698, MaxFileDescriptor=60000, SystemLoadAverage=392, ProcessCount=170, AvailableMemoryMB=5827 2023-10-27 23:52:50,264 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:52:50,273 DEBUG [Listener at localhost/43373] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-10-27 23:52:50,277 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:59288, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-10-27 23:52:50,303 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-10-27 23:52:50,303 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-10-27 23:52:50,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(492): Client=jenkins//188.40.62.62 set balanceSwitch=false 2023-10-27 23:52:50,310 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:50,313 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-10-27 23:52:50,315 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-10-27 23:52:50,318 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:50,319 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a empty. 2023-10-27 23:52:50,321 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:50,321 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionUnevenDaughter regions 2023-10-27 23:52:50,342 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/.tabledesc/.tableinfo.0000000001 2023-10-27 23:52:50,344 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1a178e5e9b9e5a90d2396ab013c5750a, NAME => 'testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionUnevenDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp 2023-10-27 23:52:50,368 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:52:50,369 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1604): Closing 1a178e5e9b9e5a90d2396ab013c5750a, disabling compactions & flushes 2023-10-27 23:52:50,369 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:50,369 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:50,369 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. after waiting 0 ms 2023-10-27 23:52:50,369 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:50,369 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:50,369 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1558): Region close journal for 1a178e5e9b9e5a90d2396ab013c5750a: 2023-10-27 23:52:50,373 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-10-27 23:52:50,376 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1698450770375"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450770375"}]},"ts":"1698450770375"} 2023-10-27 23:52:50,378 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-10-27 23:52:50,380 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-10-27 23:52:50,380 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450770380"}]},"ts":"1698450770380"} 2023-10-27 23:52:50,382 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=ENABLING in hbase:meta 2023-10-27 23:52:50,401 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-10-27 23:52:50,402 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-27 23:52:50,402 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-10-27 23:52:50,402 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-10-27 23:52:50,402 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-10-27 23:52:50,403 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1a178e5e9b9e5a90d2396ab013c5750a, ASSIGN}] 2023-10-27 23:52:50,406 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1a178e5e9b9e5a90d2396ab013c5750a, ASSIGN 2023-10-27 23:52:50,407 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1a178e5e9b9e5a90d2396ab013c5750a, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,36599,1698450767241; forceNewPlan=false, retain=false 2023-10-27 23:52:50,558 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-10-27 23:52:50,558 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1a178e5e9b9e5a90d2396ab013c5750a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:50,559 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1698450770558"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450770558"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450770558"}]},"ts":"1698450770558"} 2023-10-27 23:52:50,562 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 1a178e5e9b9e5a90d2396ab013c5750a, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:52:50,714 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:50,722 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:50,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1a178e5e9b9e5a90d2396ab013c5750a, NAME => 'testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:52:50,722 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:52:50,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:50,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:52:50,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:50,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:50,725 INFO [StoreOpener-1a178e5e9b9e5a90d2396ab013c5750a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:50,727 DEBUG [StoreOpener-1a178e5e9b9e5a90d2396ab013c5750a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1 2023-10-27 23:52:50,727 DEBUG [StoreOpener-1a178e5e9b9e5a90d2396ab013c5750a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1 2023-10-27 23:52:50,728 INFO [StoreOpener-1a178e5e9b9e5a90d2396ab013c5750a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a178e5e9b9e5a90d2396ab013c5750a columnFamilyName cf1 2023-10-27 23:52:50,729 INFO [StoreOpener-1a178e5e9b9e5a90d2396ab013c5750a-1] regionserver.HStore(310): Store=1a178e5e9b9e5a90d2396ab013c5750a/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:50,729 INFO [StoreOpener-1a178e5e9b9e5a90d2396ab013c5750a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:50,731 DEBUG [StoreOpener-1a178e5e9b9e5a90d2396ab013c5750a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2 2023-10-27 23:52:50,731 DEBUG [StoreOpener-1a178e5e9b9e5a90d2396ab013c5750a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2 2023-10-27 23:52:50,732 INFO [StoreOpener-1a178e5e9b9e5a90d2396ab013c5750a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a178e5e9b9e5a90d2396ab013c5750a columnFamilyName cf2 2023-10-27 23:52:50,733 INFO [StoreOpener-1a178e5e9b9e5a90d2396ab013c5750a-1] regionserver.HStore(310): Store=1a178e5e9b9e5a90d2396ab013c5750a/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:50,735 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:50,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:50,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:52:50,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:50,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-10-27 23:52:50,746 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 1a178e5e9b9e5a90d2396ab013c5750a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66434070, jitterRate=-0.010055214166641235}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:52:50,746 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 1a178e5e9b9e5a90d2396ab013c5750a: 2023-10-27 23:52:50,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a., pid=11, masterSystemTime=1698450770714 2023-10-27 23:52:50,750 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:50,750 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:50,751 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1a178e5e9b9e5a90d2396ab013c5750a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:50,751 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1698450770751"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450770751"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450770751"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450770751"}]},"ts":"1698450770751"} 2023-10-27 23:52:50,759 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=11, resume processing ppid=10 2023-10-27 23:52:50,759 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 1a178e5e9b9e5a90d2396ab013c5750a, server=jenkins-hbase16.apache.org,36599,1698450767241 in 193 msec 2023-10-27 23:52:50,763 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=10, resume processing ppid=9 2023-10-27 23:52:50,763 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1a178e5e9b9e5a90d2396ab013c5750a, ASSIGN in 357 msec 2023-10-27 23:52:50,765 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-10-27 23:52:50,765 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450770765"}]},"ts":"1698450770765"} 2023-10-27 23:52:50,767 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=ENABLED in hbase:meta 2023-10-27 23:52:50,818 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-10-27 23:52:50,821 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter in 510 msec 2023-10-27 23:52:51,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:52:51,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,096 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-10-27 23:52:51,096 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:51,097 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-10-27 23:52:51,097 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-10-27 23:52:51,097 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,097 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,108 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,108 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,108 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,108 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,109 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,109 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:52:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:51,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,112 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:51,112 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:51,113 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:51,113 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:51,113 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:51,113 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:51,113 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:51,113 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:51,114 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:51,114 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:51,114 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:51,114 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:51,114 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:51,115 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:52:51,115 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:51,115 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:52:51,116 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. started... 2023-10-27 23:52:51,117 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a178e5e9b9e5a90d2396ab013c5750a 2/2 column families, dataSize=120 B heapSize=928 B 2023-10-27 23:52:51,125 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,125 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,125 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,125 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,125 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:51,125 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:51,194 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/68b4536a221a42628ce69ded1e41e018 2023-10-27 23:52:51,256 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/dd03d244abfe47159e74a09813bcaec7 2023-10-27 23:52:51,267 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/68b4536a221a42628ce69ded1e41e018 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/68b4536a221a42628ce69ded1e41e018 2023-10-27 23:52:51,277 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/68b4536a221a42628ce69ded1e41e018, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:52:51,281 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/dd03d244abfe47159e74a09813bcaec7 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/dd03d244abfe47159e74a09813bcaec7 2023-10-27 23:52:51,291 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/dd03d244abfe47159e74a09813bcaec7, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:52:51,293 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 1a178e5e9b9e5a90d2396ab013c5750a in 176ms, sequenceid=6, compaction requested=false 2023-10-27 23:52:51,295 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionUnevenDaughter' 2023-10-27 23:52:51,297 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a178e5e9b9e5a90d2396ab013c5750a: 2023-10-27 23:52:51,297 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:51,297 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:52:51,298 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:52:51,298 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,298 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:51,298 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:51,308 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,308 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:51,309 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,309 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:51,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-10-27 23:52:51,312 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-10-27 23:52:51,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f4e3811[Count = 0] remaining members to acquire global barrier 2023-10-27 23:52:51,312 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,341 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,341 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,341 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,341 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,341 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,342 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:52:51,342 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:51,341 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:51,342 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:51,342 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:51,342 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:51,342 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:52:51,342 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:52:51,342 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:52:51,355 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:51,355 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:51,355 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:51,361 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:51,361 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:51,362 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:51,361 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,362 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:51,362 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:51,363 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase16.apache.org,36599,1698450767241': 2023-10-27 23:52:51,368 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-10-27 23:52:51,368 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:52:51,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:52:51,368 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:52:51,369 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,369 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:52:51,378 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,378 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,378 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:51,378 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,378 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:51,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,378 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,379 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,379 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:51,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:51,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:51,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:51,380 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,380 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:51,382 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,382 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,383 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,400 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,400 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,400 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,400 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,400 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,400 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:51,400 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,400 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-10-27 23:52:51,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:52:51,401 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:51,400 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,400 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:51,401 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,403 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,404 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,404 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,404 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,404 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:51,404 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:51,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:51,406 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:52:51,406 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:52:51,506 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:52:51,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:52:51,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:52:51,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:52:51,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,544 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-10-27 23:52:51,544 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:51,544 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-10-27 23:52:51,544 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-10-27 23:52:51,545 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,545 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,553 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,553 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,553 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,553 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,554 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,554 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:52:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:51,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,555 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:51,555 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:51,556 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:51,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:51,556 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:51,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:51,558 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:51,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,558 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:51,558 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:51,558 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:51,558 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:51,559 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:51,559 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:51,559 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:51,559 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:51,559 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:52:51,560 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:52:51,560 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:51,560 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. started... 2023-10-27 23:52:51,561 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a178e5e9b9e5a90d2396ab013c5750a 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:52:51,570 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,579 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,579 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,580 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:51,581 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,581 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:51,584 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/5d07fa5f8b3d4b33818f9c523db84071 2023-10-27 23:52:51,613 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/04ba4141a0f34d8eb85680b2928c30b8 2023-10-27 23:52:51,622 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/5d07fa5f8b3d4b33818f9c523db84071 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/5d07fa5f8b3d4b33818f9c523db84071 2023-10-27 23:52:51,632 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/5d07fa5f8b3d4b33818f9c523db84071, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:52:51,633 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/04ba4141a0f34d8eb85680b2928c30b8 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/04ba4141a0f34d8eb85680b2928c30b8 2023-10-27 23:52:51,643 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/04ba4141a0f34d8eb85680b2928c30b8, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:52:51,644 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1a178e5e9b9e5a90d2396ab013c5750a in 84ms, sequenceid=19, compaction requested=false 2023-10-27 23:52:51,645 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a178e5e9b9e5a90d2396ab013c5750a: 2023-10-27 23:52:51,645 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:51,645 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:52:51,645 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:52:51,645 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,646 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:51,646 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:51,653 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,653 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:51,654 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,654 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:51,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-10-27 23:52:51,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48258b9e[Count = 0] remaining members to acquire global barrier 2023-10-27 23:52:51,656 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-10-27 23:52:51,656 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,666 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,666 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,666 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,667 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:51,667 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,667 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:52:51,667 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:51,667 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:51,667 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:52:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,667 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:51,667 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:52:51,667 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:51,667 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:51,667 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:52:51,679 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:51,679 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:51,679 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:51,686 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:51,686 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,686 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,687 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:51,686 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:51,687 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:51,687 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase16.apache.org,36599,1698450767241': 2023-10-27 23:52:51,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:52:51,694 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-10-27 23:52:51,694 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:52:51,694 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:52:51,694 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,694 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:52:51,703 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,703 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,703 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:51,703 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,703 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:51,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,703 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:51,704 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,704 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:51,704 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:51,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:51,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:51,705 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,706 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,707 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,707 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:51,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:51,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter because node does not exist (not an error) 2023-10-27 23:52:51,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-10-27 23:52:51,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:52:51,738 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,739 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:51,739 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:51,739 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:51,739 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:52:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:51,739 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:51,740 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,739 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:52:51,740 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,840 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:52:51,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:52:51,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:52:51,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:52:51,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,870 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-10-27 23:52:51,870 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:51,871 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-10-27 23:52:51,871 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-10-27 23:52:51,872 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,872 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,883 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,883 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,883 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,883 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:51,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:51,884 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,884 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:52:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:51,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:51,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:51,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,885 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:51,886 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:51,885 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:51,886 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:51,886 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:51,886 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:51,886 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:51,887 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:51,887 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:51,887 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:51,887 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:51,888 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:51,888 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:51,888 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:52:51,888 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:51,888 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:52:51,888 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. started... 2023-10-27 23:52:51,888 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a178e5e9b9e5a90d2396ab013c5750a 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:52:51,895 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,906 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,906 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,907 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:51,907 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,907 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:51,912 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/a88b5a60a38644eaa0012136ec701fc0 2023-10-27 23:52:51,934 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/16dc81cf0aa24a738be12a15678ccb7f 2023-10-27 23:52:51,944 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/a88b5a60a38644eaa0012136ec701fc0 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/a88b5a60a38644eaa0012136ec701fc0 2023-10-27 23:52:51,953 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/a88b5a60a38644eaa0012136ec701fc0, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:52:51,955 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/16dc81cf0aa24a738be12a15678ccb7f as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/16dc81cf0aa24a738be12a15678ccb7f 2023-10-27 23:52:51,964 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/16dc81cf0aa24a738be12a15678ccb7f, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:52:51,966 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1a178e5e9b9e5a90d2396ab013c5750a in 78ms, sequenceid=32, compaction requested=true 2023-10-27 23:52:51,966 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a178e5e9b9e5a90d2396ab013c5750a: 2023-10-27 23:52:51,966 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:51,966 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:52:51,967 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:52:51,967 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,967 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:51,967 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:51,978 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,978 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:51,979 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,979 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:51,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-10-27 23:52:51,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7caf33d3[Count = 0] remaining members to acquire global barrier 2023-10-27 23:52:51,981 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-10-27 23:52:51,981 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,991 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,991 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,991 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,992 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:51,992 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:51,992 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:51,991 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:51,992 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:52:51,992 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:52:51,992 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:51,992 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:52:51,992 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:51,992 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:51,992 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:52:52,001 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:52,001 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:52,001 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:52,011 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:52,012 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:52,012 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:52,011 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:52,020 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:52,020 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:52,020 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase16.apache.org,36599,1698450767241': 2023-10-27 23:52:52,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:52:52,025 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-10-27 23:52:52,025 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:52:52,025 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:52:52,025 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,025 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:52:52,036 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,037 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,037 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,037 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,037 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,037 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,037 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:52,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,037 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:52,038 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,038 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:52,041 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,041 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,042 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,053 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,053 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,053 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,053 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,053 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:52,053 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,053 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,053 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,054 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-10-27 23:52:52,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:52:52,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,053 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,054 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,055 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,055 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:52:52,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,055 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,055 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:52:52,055 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,055 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,055 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,055 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,055 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,155 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:52:52,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:52:52,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:52:52,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:52:52,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,187 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-10-27 23:52:52,187 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:52,188 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-10-27 23:52:52,188 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-10-27 23:52:52,189 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,189 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,195 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,195 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,195 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,195 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,195 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,195 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:52:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:52,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:52,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:52,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,197 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:52,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,197 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:52,197 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:52,198 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:52,197 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:52,198 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:52,198 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:52,198 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:52:52,199 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:52,199 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:52:52,199 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:52,199 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:52,198 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:52,199 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:52,199 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:52,199 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:52,199 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. started... 2023-10-27 23:52:52,200 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a178e5e9b9e5a90d2396ab013c5750a 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:52:52,208 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,219 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,219 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:52,220 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,221 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,221 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:52,223 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/10ad920c9db24c04afaf6e5823d8604a 2023-10-27 23:52:52,282 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/f3d88e7ea44640e098237e6df13ca0e0 2023-10-27 23:52:52,292 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/10ad920c9db24c04afaf6e5823d8604a as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/10ad920c9db24c04afaf6e5823d8604a 2023-10-27 23:52:52,300 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/10ad920c9db24c04afaf6e5823d8604a, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:52:52,301 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/f3d88e7ea44640e098237e6df13ca0e0 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/f3d88e7ea44640e098237e6df13ca0e0 2023-10-27 23:52:52,310 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/f3d88e7ea44640e098237e6df13ca0e0, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:52:52,311 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1a178e5e9b9e5a90d2396ab013c5750a in 111ms, sequenceid=45, compaction requested=true 2023-10-27 23:52:52,311 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a178e5e9b9e5a90d2396ab013c5750a: 2023-10-27 23:52:52,311 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:52,311 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:52:52,312 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:52:52,312 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,312 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:52,312 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:52,339 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,339 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:52,339 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,339 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:52,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-10-27 23:52:52,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@482bbc41[Count = 0] remaining members to acquire global barrier 2023-10-27 23:52:52,342 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-10-27 23:52:52,342 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,353 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,353 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,353 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,353 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:52,353 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:52,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,353 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,354 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:52:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,353 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:52:52,354 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,354 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:52,354 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:52,354 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:52:52,354 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:52,354 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:52:52,361 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:52,361 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,362 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:52,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:52,362 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:52,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:52,370 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:52,370 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:52,370 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:52,378 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:52,378 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:52,378 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase16.apache.org,36599,1698450767241': 2023-10-27 23:52:52,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:52:52,383 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-10-27 23:52:52,383 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:52:52,383 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:52:52,383 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,383 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:52:52,391 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,391 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,391 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,391 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,392 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,392 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,391 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,392 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:52,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,392 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,393 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,394 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:52,395 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,395 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,403 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,403 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,403 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,403 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:52,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-10-27 23:52:52,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:52:52,403 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,403 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,403 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,403 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,405 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,405 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,405 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:52:52,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,405 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,405 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:52:52,405 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,405 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,405 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,406 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,406 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,406 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,406 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,505 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:52:52,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:52:52,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:52:52,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:52:52,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,535 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-10-27 23:52:52,535 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:52,536 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-10-27 23:52:52,536 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-10-27 23:52:52,536 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,536 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,545 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,545 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,545 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,545 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,545 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,545 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:52:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:52,546 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:52,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:52,547 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:52,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,547 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:52,548 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:52,548 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:52,547 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:52,548 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:52,548 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:52,548 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:52,548 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:52,548 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:52,550 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:52,550 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:52,550 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:52:52,550 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:52:52,550 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:52,550 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. started... 2023-10-27 23:52:52,550 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a178e5e9b9e5a90d2396ab013c5750a 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:52:52,558 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,566 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,566 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:52,567 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,567 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/9a6eb09cb9524fe7943e1ce637ff1842 2023-10-27 23:52:52,567 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,567 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:52,601 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/b117b97848c047f18901f0277d2b9494 2023-10-27 23:52:52,610 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/9a6eb09cb9524fe7943e1ce637ff1842 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/9a6eb09cb9524fe7943e1ce637ff1842 2023-10-27 23:52:52,617 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/9a6eb09cb9524fe7943e1ce637ff1842, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:52:52,619 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/b117b97848c047f18901f0277d2b9494 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/b117b97848c047f18901f0277d2b9494 2023-10-27 23:52:52,626 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/b117b97848c047f18901f0277d2b9494, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:52:52,628 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1a178e5e9b9e5a90d2396ab013c5750a in 78ms, sequenceid=58, compaction requested=true 2023-10-27 23:52:52,628 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a178e5e9b9e5a90d2396ab013c5750a: 2023-10-27 23:52:52,628 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:52,628 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:52:52,630 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:52:52,630 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,630 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:52,630 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:52,636 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,636 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:52,638 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,638 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:52,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-10-27 23:52:52,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26cb9be5[Count = 0] remaining members to acquire global barrier 2023-10-27 23:52:52,640 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-10-27 23:52:52,640 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,650 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,650 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,650 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,650 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:52,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,650 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:52,650 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,650 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:52:52,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,650 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:52:52,650 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,650 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:52,650 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:52,650 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:52:52,651 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:52,651 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:52:52,658 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:52,658 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:52,658 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:52,666 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:52,666 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:52,666 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,666 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:52,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,666 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:52,667 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:52,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:52,666 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase16.apache.org,36599,1698450767241': 2023-10-27 23:52:52,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:52:52,672 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-10-27 23:52:52,672 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:52:52,672 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:52:52,672 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,672 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:52:52,683 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,683 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,683 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,683 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,683 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,683 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,683 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,684 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,684 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,685 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,686 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:52,686 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,687 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,714 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,714 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,714 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,714 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,715 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,715 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:52,715 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,715 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,715 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-10-27 23:52:52,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:52:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,715 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:52,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,716 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,716 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,717 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:52:52,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:52,717 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,717 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:52:52,717 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,717 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,717 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,717 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,717 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,817 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:52:52,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:52:52,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:52:52,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:52:52,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,846 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-10-27 23:52:52,846 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:52,847 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-10-27 23:52:52,847 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-10-27 23:52:52,848 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,848 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,858 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,858 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,858 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,858 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:52,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,858 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,859 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:52:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:52,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:52,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:52,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:52,860 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:52,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-10-27 23:52:52,860 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:52,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,860 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:52,861 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-10-27 23:52:52,861 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:52,861 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:52,861 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:52,861 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:52,861 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:52,861 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:52,862 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-10-27 23:52:52,862 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:52,862 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:52,862 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:52,862 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:52:52,862 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:52:52,862 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. started... 2023-10-27 23:52:52,862 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a178e5e9b9e5a90d2396ab013c5750a 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:52:52,870 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,870 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,870 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,870 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,870 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:52,870 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:52,883 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/e8f365c77ec64b85ac82f587706cc82e 2023-10-27 23:52:52,907 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/53c3985a23aa4a0cbbe83ea9449370ec 2023-10-27 23:52:52,916 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/e8f365c77ec64b85ac82f587706cc82e as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/e8f365c77ec64b85ac82f587706cc82e 2023-10-27 23:52:52,923 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/e8f365c77ec64b85ac82f587706cc82e, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:52:52,925 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/53c3985a23aa4a0cbbe83ea9449370ec as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/53c3985a23aa4a0cbbe83ea9449370ec 2023-10-27 23:52:52,933 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/53c3985a23aa4a0cbbe83ea9449370ec, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:52:52,934 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1a178e5e9b9e5a90d2396ab013c5750a in 72ms, sequenceid=71, compaction requested=true 2023-10-27 23:52:52,935 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a178e5e9b9e5a90d2396ab013c5750a: 2023-10-27 23:52:52,935 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:52,935 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:52:52,935 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:52:52,935 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,936 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-10-27 23:52:52,936 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-10-27 23:52:52,945 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,945 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:52,945 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,945 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:52,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:52,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:52,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:52,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:52,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-10-27 23:52:52,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42568dd9[Count = 0] remaining members to acquire global barrier 2023-10-27 23:52:52,947 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-10-27 23:52:52,947 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,041 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,041 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,041 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,042 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:53,042 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,042 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:52:53,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,042 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:53,042 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:53,042 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:52:53,042 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,042 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:52:53,042 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-10-27 23:52:53,043 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-10-27 23:52:53,043 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:52:53,143 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:53,143 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:53,143 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:53,191 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:53,192 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:53,192 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,191 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-10-27 23:52:53,193 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:53,192 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:53,193 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-10-27 23:52:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase16.apache.org,36599,1698450767241': 2023-10-27 23:52:53,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:52:53,197 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-10-27 23:52:53,197 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:52:53,198 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:52:53,198 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,198 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:52:53,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:53,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:53,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,367 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:53,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:53,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,367 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:53,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,368 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,369 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:53,370 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:53,370 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:53,510 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:53,510 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:53,510 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:53,511 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:53,510 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:53,510 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:53,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-10-27 23:52:53,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:52:53,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:53,511 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:53,511 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:53,512 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:53,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:53,512 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:53,512 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:52:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:53,512 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:53,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:53,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:53,512 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:52:53,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:53,512 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,513 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:53,513 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,513 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,513 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:53,513 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,513 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-10-27 23:52:53,613 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:52:53,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:52:53,638 DEBUG [Listener at localhost/43373] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,645 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:53,646 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:53,646 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:53,646 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:53,653 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1a178e5e9b9e5a90d2396ab013c5750a, UNASSIGN}] 2023-10-27 23:52:53,655 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1a178e5e9b9e5a90d2396ab013c5750a, UNASSIGN 2023-10-27 23:52:53,656 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1a178e5e9b9e5a90d2396ab013c5750a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,656 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1698450773656"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450773656"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450773656"}]},"ts":"1698450773656"} 2023-10-27 23:52:53,659 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 1a178e5e9b9e5a90d2396ab013c5750a, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:52:53,815 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:53,819 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,820 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 1a178e5e9b9e5a90d2396ab013c5750a, disabling compactions & flushes 2023-10-27 23:52:53,820 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:53,820 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:53,820 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. after waiting 0 ms 2023-10-27 23:52:53,820 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:53,820 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 1a178e5e9b9e5a90d2396ab013c5750a 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-10-27 23:52:53,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/aaa71348029d4cc993f752bafef54c2e 2023-10-27 23:52:53,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/e2db600972e34421ab3a9ebd4e3f8ba0 2023-10-27 23:52:53,871 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf1/aaa71348029d4cc993f752bafef54c2e as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/aaa71348029d4cc993f752bafef54c2e 2023-10-27 23:52:53,879 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/aaa71348029d4cc993f752bafef54c2e, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:52:53,881 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.tmp/cf2/e2db600972e34421ab3a9ebd4e3f8ba0 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/e2db600972e34421ab3a9ebd4e3f8ba0 2023-10-27 23:52:53,891 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/e2db600972e34421ab3a9ebd4e3f8ba0, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:52:53,892 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 1a178e5e9b9e5a90d2396ab013c5750a in 72ms, sequenceid=82, compaction requested=true 2023-10-27 23:52:53,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-10-27 23:52:53,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:52:53,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. 2023-10-27 23:52:53,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 1a178e5e9b9e5a90d2396ab013c5750a: 2023-10-27 23:52:53,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,920 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1a178e5e9b9e5a90d2396ab013c5750a, regionState=CLOSED 2023-10-27 23:52:53,921 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1698450773920"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450773920"}]},"ts":"1698450773920"} 2023-10-27 23:52:53,925 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=14, resume processing ppid=13 2023-10-27 23:52:53,925 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 1a178e5e9b9e5a90d2396ab013c5750a, server=jenkins-hbase16.apache.org,36599,1698450767241 in 264 msec 2023-10-27 23:52:53,928 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=13, resume processing ppid=12 2023-10-27 23:52:53,928 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1a178e5e9b9e5a90d2396ab013c5750a, UNASSIGN in 273 msec 2023-10-27 23:52:53,948 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=12 splitting 14 storefiles, region=1a178e5e9b9e5a90d2396ab013c5750a, threads=14 2023-10-27 23:52:53,949 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/04ba4141a0f34d8eb85680b2928c30b8 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,949 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/16dc81cf0aa24a738be12a15678ccb7f for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,949 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/53c3985a23aa4a0cbbe83ea9449370ec for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,950 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/b117b97848c047f18901f0277d2b9494 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,950 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/e2db600972e34421ab3a9ebd4e3f8ba0 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,950 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/f3d88e7ea44640e098237e6df13ca0e0 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,950 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/dd03d244abfe47159e74a09813bcaec7 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,950 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/10ad920c9db24c04afaf6e5823d8604a for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,950 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/5d07fa5f8b3d4b33818f9c523db84071 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,951 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/68b4536a221a42628ce69ded1e41e018 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,951 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/9a6eb09cb9524fe7943e1ce637ff1842 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,955 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/a88b5a60a38644eaa0012136ec701fc0 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,955 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/aaa71348029d4cc993f752bafef54c2e for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,955 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/e8f365c77ec64b85ac82f587706cc82e for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:53,973 DEBUG [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/53c3985a23aa4a0cbbe83ea9449370ec, top=true 2023-10-27 23:52:53,986 DEBUG [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/9a6eb09cb9524fe7943e1ce637ff1842, top=true 2023-10-27 23:52:53,989 DEBUG [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/e2db600972e34421ab3a9ebd4e3f8ba0, top=true 2023-10-27 23:52:53,998 DEBUG [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/b117b97848c047f18901f0277d2b9494, top=true 2023-10-27 23:52:54,025 INFO [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.splits/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-53c3985a23aa4a0cbbe83ea9449370ec for child: 9aca977d0db042f340c741ee9446ec83, parent: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,025 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/53c3985a23aa4a0cbbe83ea9449370ec for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,026 INFO [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.splits/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-9a6eb09cb9524fe7943e1ce637ff1842 for child: 9aca977d0db042f340c741ee9446ec83, parent: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,026 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/9a6eb09cb9524fe7943e1ce637ff1842 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,027 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/aaa71348029d4cc993f752bafef54c2e, top=true 2023-10-27 23:52:54,027 DEBUG [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/e8f365c77ec64b85ac82f587706cc82e, top=true 2023-10-27 23:52:54,041 INFO [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.splits/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-b117b97848c047f18901f0277d2b9494 for child: 9aca977d0db042f340c741ee9446ec83, parent: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,041 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/b117b97848c047f18901f0277d2b9494 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,057 INFO [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.splits/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e2db600972e34421ab3a9ebd4e3f8ba0 for child: 9aca977d0db042f340c741ee9446ec83, parent: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,057 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/e2db600972e34421ab3a9ebd4e3f8ba0 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,067 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.splits/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-aaa71348029d4cc993f752bafef54c2e for child: 9aca977d0db042f340c741ee9446ec83, parent: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,067 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/aaa71348029d4cc993f752bafef54c2e for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,067 INFO [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/.splits/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e8f365c77ec64b85ac82f587706cc82e for child: 9aca977d0db042f340c741ee9446ec83, parent: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,067 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/e8f365c77ec64b85ac82f587706cc82e for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,139 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/68b4536a221a42628ce69ded1e41e018 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,140 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/f3d88e7ea44640e098237e6df13ca0e0 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,163 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/dd03d244abfe47159e74a09813bcaec7 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,175 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/04ba4141a0f34d8eb85680b2928c30b8 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,178 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/5d07fa5f8b3d4b33818f9c523db84071 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,507 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/a88b5a60a38644eaa0012136ec701fc0 for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,507 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/10ad920c9db24c04afaf6e5823d8604a for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,520 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/16dc81cf0aa24a738be12a15678ccb7f for region: 1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:54,520 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=12 split storefiles for region 1a178e5e9b9e5a90d2396ab013c5750a Daughter A: 8 storefiles, Daughter B: 14 storefiles. 2023-10-27 23:52:54,571 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-10-27 23:52:54,573 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-10-27 23:52:54,576 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1698450774576"},{"qualifier":"splitA","vlen":70,"tag":[],"timestamp":"1698450774576"},{"qualifier":"splitB","vlen":70,"tag":[],"timestamp":"1698450774576"}]},"ts":"1698450774576"} 2023-10-27 23:52:54,576 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1698450774576"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450774576"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450774576"}]},"ts":"1698450774576"} 2023-10-27 23:52:54,577 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1698450774576"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450774576"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450774576"}]},"ts":"1698450774576"} 2023-10-27 23:52:54,619 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=8285b45cae27573bc22fb00c23a6d6fe, ASSIGN}, {pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=9aca977d0db042f340c741ee9446ec83, ASSIGN}] 2023-10-27 23:52:54,620 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=9aca977d0db042f340c741ee9446ec83, ASSIGN 2023-10-27 23:52:54,621 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=9aca977d0db042f340c741ee9446ec83, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase16.apache.org,36599,1698450767241; forceNewPlan=false, retain=false 2023-10-27 23:52:54,622 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=8285b45cae27573bc22fb00c23a6d6fe, ASSIGN 2023-10-27 23:52:54,623 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=8285b45cae27573bc22fb00c23a6d6fe, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase16.apache.org,36599,1698450767241; forceNewPlan=false, retain=false 2023-10-27 23:52:54,689 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-27 23:52:54,772 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-10-27 23:52:54,773 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=8285b45cae27573bc22fb00c23a6d6fe, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:54,774 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1698450774773"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450774773"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450774773"}]},"ts":"1698450774773"} 2023-10-27 23:52:54,776 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=17, ppid=15, state=RUNNABLE; OpenRegionProcedure 8285b45cae27573bc22fb00c23a6d6fe, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:52:54,780 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-10-27 23:52:54,781 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-10-27 23:52:54,782 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=9aca977d0db042f340c741ee9446ec83, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:54,783 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1698450774782"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450774782"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450774782"}]},"ts":"1698450774782"} 2023-10-27 23:52:54,785 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=18, ppid=16, state=RUNNABLE; OpenRegionProcedure 9aca977d0db042f340c741ee9446ec83, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:52:54,930 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:54,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe. 2023-10-27 23:52:54,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8285b45cae27573bc22fb00c23a6d6fe, NAME => 'testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.', STARTKEY => '', ENDKEY => '26'} 2023-10-27 23:52:54,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:52:54,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter 8285b45cae27573bc22fb00c23a6d6fe 2023-10-27 23:52:54,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:52:54,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 8285b45cae27573bc22fb00c23a6d6fe 2023-10-27 23:52:54,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 8285b45cae27573bc22fb00c23a6d6fe 2023-10-27 23:52:54,938 INFO [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 8285b45cae27573bc22fb00c23a6d6fe 2023-10-27 23:52:54,939 DEBUG [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1 2023-10-27 23:52:54,939 DEBUG [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1 2023-10-27 23:52:54,939 INFO [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8285b45cae27573bc22fb00c23a6d6fe columnFamilyName cf1 2023-10-27 23:52:54,974 DEBUG [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/10ad920c9db24c04afaf6e5823d8604a.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/10ad920c9db24c04afaf6e5823d8604a-bottom 2023-10-27 23:52:54,983 DEBUG [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/5d07fa5f8b3d4b33818f9c523db84071.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/5d07fa5f8b3d4b33818f9c523db84071-bottom 2023-10-27 23:52:54,993 DEBUG [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/68b4536a221a42628ce69ded1e41e018.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/68b4536a221a42628ce69ded1e41e018-bottom 2023-10-27 23:52:55,002 DEBUG [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/a88b5a60a38644eaa0012136ec701fc0.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/a88b5a60a38644eaa0012136ec701fc0-bottom 2023-10-27 23:52:55,003 INFO [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] regionserver.HStore(310): Store=8285b45cae27573bc22fb00c23a6d6fe/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:55,003 INFO [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 8285b45cae27573bc22fb00c23a6d6fe 2023-10-27 23:52:55,004 DEBUG [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2 2023-10-27 23:52:55,004 DEBUG [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2 2023-10-27 23:52:55,005 INFO [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8285b45cae27573bc22fb00c23a6d6fe columnFamilyName cf2 2023-10-27 23:52:55,026 DEBUG [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/04ba4141a0f34d8eb85680b2928c30b8.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/04ba4141a0f34d8eb85680b2928c30b8-bottom 2023-10-27 23:52:55,034 DEBUG [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/16dc81cf0aa24a738be12a15678ccb7f.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/16dc81cf0aa24a738be12a15678ccb7f-bottom 2023-10-27 23:52:55,041 DEBUG [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/dd03d244abfe47159e74a09813bcaec7.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/dd03d244abfe47159e74a09813bcaec7-bottom 2023-10-27 23:52:55,048 DEBUG [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/f3d88e7ea44640e098237e6df13ca0e0.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/f3d88e7ea44640e098237e6df13ca0e0-bottom 2023-10-27 23:52:55,049 INFO [StoreOpener-8285b45cae27573bc22fb00c23a6d6fe-1] regionserver.HStore(310): Store=8285b45cae27573bc22fb00c23a6d6fe/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:55,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe 2023-10-27 23:52:55,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe 2023-10-27 23:52:55,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:52:55,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 8285b45cae27573bc22fb00c23a6d6fe 2023-10-27 23:52:55,056 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 8285b45cae27573bc22fb00c23a6d6fe; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63235353, jitterRate=-0.05771981179714203}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:52:55,056 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 8285b45cae27573bc22fb00c23a6d6fe: 2023-10-27 23:52:55,057 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe., pid=17, masterSystemTime=1698450774930 2023-10-27 23:52:55,059 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:52:55,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:52:55,060 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-10-27 23:52:55,060 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-10-27 23:52:55,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe. 2023-10-27 23:52:55,062 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe. 2023-10-27 23:52:55,062 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe. 2023-10-27 23:52:55,063 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1912): 8285b45cae27573bc22fb00c23a6d6fe/cf1 is initiating minor compaction (all files) 2023-10-27 23:52:55,063 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe. 2023-10-27 23:52:55,063 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8285b45cae27573bc22fb00c23a6d6fe/cf1 in testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe. 2023-10-27 23:52:55,063 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83. 2023-10-27 23:52:55,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9aca977d0db042f340c741ee9446ec83, NAME => 'testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.', STARTKEY => '26', ENDKEY => ''} 2023-10-27 23:52:55,063 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=8285b45cae27573bc22fb00c23a6d6fe, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:55,063 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/68b4536a221a42628ce69ded1e41e018.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/68b4536a221a42628ce69ded1e41e018-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/5d07fa5f8b3d4b33818f9c523db84071.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/5d07fa5f8b3d4b33818f9c523db84071-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/a88b5a60a38644eaa0012136ec701fc0.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/a88b5a60a38644eaa0012136ec701fc0-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/10ad920c9db24c04afaf6e5823d8604a.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/10ad920c9db24c04afaf6e5823d8604a-bottom] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/.tmp, totalSize=19.9 K 2023-10-27 23:52:55,063 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1698450775063"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450775063"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450775063"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450775063"}]},"ts":"1698450775063"} 2023-10-27 23:52:55,064 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:52:55,064 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1912): 8285b45cae27573bc22fb00c23a6d6fe/cf2 is initiating minor compaction (all files) 2023-10-27 23:52:55,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter 9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:55,065 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8285b45cae27573bc22fb00c23a6d6fe/cf2 in testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe. 2023-10-27 23:52:55,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:52:55,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:55,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:55,066 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/dd03d244abfe47159e74a09813bcaec7.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/dd03d244abfe47159e74a09813bcaec7-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/04ba4141a0f34d8eb85680b2928c30b8.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/04ba4141a0f34d8eb85680b2928c30b8-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/16dc81cf0aa24a738be12a15678ccb7f.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/16dc81cf0aa24a738be12a15678ccb7f-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/f3d88e7ea44640e098237e6df13ca0e0.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/f3d88e7ea44640e098237e6df13ca0e0-bottom] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/.tmp, totalSize=19.9 K 2023-10-27 23:52:55,066 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 68b4536a221a42628ce69ded1e41e018.1a178e5e9b9e5a90d2396ab013c5750a, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450771066 2023-10-27 23:52:55,067 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting dd03d244abfe47159e74a09813bcaec7.1a178e5e9b9e5a90d2396ab013c5750a, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450771066 2023-10-27 23:52:55,068 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 5d07fa5f8b3d4b33818f9c523db84071.1a178e5e9b9e5a90d2396ab013c5750a, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450771513 2023-10-27 23:52:55,068 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting 04ba4141a0f34d8eb85680b2928c30b8.1a178e5e9b9e5a90d2396ab013c5750a, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450771513 2023-10-27 23:52:55,069 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting a88b5a60a38644eaa0012136ec701fc0.1a178e5e9b9e5a90d2396ab013c5750a, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450771842 2023-10-27 23:52:55,069 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting 16dc81cf0aa24a738be12a15678ccb7f.1a178e5e9b9e5a90d2396ab013c5750a, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450771842 2023-10-27 23:52:55,070 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 10ad920c9db24c04afaf6e5823d8604a.1a178e5e9b9e5a90d2396ab013c5750a, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450772158 2023-10-27 23:52:55,070 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting f3d88e7ea44640e098237e6df13ca0e0.1a178e5e9b9e5a90d2396ab013c5750a, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450772158 2023-10-27 23:52:55,072 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=17, resume processing ppid=15 2023-10-27 23:52:55,072 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=17, ppid=15, state=SUCCESS; OpenRegionProcedure 8285b45cae27573bc22fb00c23a6d6fe, server=jenkins-hbase16.apache.org,36599,1698450767241 in 292 msec 2023-10-27 23:52:55,075 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=8285b45cae27573bc22fb00c23a6d6fe, ASSIGN in 453 msec 2023-10-27 23:52:55,077 INFO [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:55,079 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1 2023-10-27 23:52:55,079 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1 2023-10-27 23:52:55,080 INFO [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9aca977d0db042f340c741ee9446ec83 columnFamilyName cf1 2023-10-27 23:52:55,093 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8285b45cae27573bc22fb00c23a6d6fe#cf1#compaction#14 average throughput is 0.21 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:52:55,096 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] throttle.PressureAwareThroughputController(145): 8285b45cae27573bc22fb00c23a6d6fe#cf2#compaction#15 average throughput is 0.43 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:52:55,105 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/10ad920c9db24c04afaf6e5823d8604a.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/10ad920c9db24c04afaf6e5823d8604a-top 2023-10-27 23:52:55,110 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/5d07fa5f8b3d4b33818f9c523db84071.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/5d07fa5f8b3d4b33818f9c523db84071-top 2023-10-27 23:52:55,117 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/68b4536a221a42628ce69ded1e41e018.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/68b4536a221a42628ce69ded1e41e018-top 2023-10-27 23:52:55,125 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/a88b5a60a38644eaa0012136ec701fc0.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/a88b5a60a38644eaa0012136ec701fc0-top 2023-10-27 23:52:55,128 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/.tmp/cf1/a57b39a149744799888f8b8b817a11ec as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/a57b39a149744799888f8b8b817a11ec 2023-10-27 23:52:55,131 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/.tmp/cf2/068c1f976790455bb5f77d7e1a552f36 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/068c1f976790455bb5f77d7e1a552f36 2023-10-27 23:52:55,134 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-9a6eb09cb9524fe7943e1ce637ff1842 2023-10-27 23:52:55,139 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-aaa71348029d4cc993f752bafef54c2e 2023-10-27 23:52:55,146 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e8f365c77ec64b85ac82f587706cc82e 2023-10-27 23:52:55,147 INFO [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(310): Store=9aca977d0db042f340c741ee9446ec83/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:55,147 INFO [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:55,148 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8285b45cae27573bc22fb00c23a6d6fe/cf2 of 8285b45cae27573bc22fb00c23a6d6fe into 068c1f976790455bb5f77d7e1a552f36(size=5.5 K), total size for store is 5.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:52:55,148 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8285b45cae27573bc22fb00c23a6d6fe/cf1 of 8285b45cae27573bc22fb00c23a6d6fe into a57b39a149744799888f8b8b817a11ec(size=5.5 K), total size for store is 5.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:52:55,149 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8285b45cae27573bc22fb00c23a6d6fe: 2023-10-27 23:52:55,149 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8285b45cae27573bc22fb00c23a6d6fe: 2023-10-27 23:52:55,149 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe., storeName=8285b45cae27573bc22fb00c23a6d6fe/cf2, priority=12, startTime=1698450775060; duration=0sec 2023-10-27 23:52:55,149 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe., storeName=8285b45cae27573bc22fb00c23a6d6fe/cf1, priority=12, startTime=1698450775059; duration=0sec 2023-10-27 23:52:55,150 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2 2023-10-27 23:52:55,150 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2 2023-10-27 23:52:55,150 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:52:55,150 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:52:55,150 INFO [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9aca977d0db042f340c741ee9446ec83 columnFamilyName cf2 2023-10-27 23:52:55,166 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/04ba4141a0f34d8eb85680b2928c30b8.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/04ba4141a0f34d8eb85680b2928c30b8-top 2023-10-27 23:52:55,172 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/16dc81cf0aa24a738be12a15678ccb7f.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/16dc81cf0aa24a738be12a15678ccb7f-top 2023-10-27 23:52:55,178 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/dd03d244abfe47159e74a09813bcaec7.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/dd03d244abfe47159e74a09813bcaec7-top 2023-10-27 23:52:55,184 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/f3d88e7ea44640e098237e6df13ca0e0.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/f3d88e7ea44640e098237e6df13ca0e0-top 2023-10-27 23:52:55,190 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-53c3985a23aa4a0cbbe83ea9449370ec 2023-10-27 23:52:55,196 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-b117b97848c047f18901f0277d2b9494 2023-10-27 23:52:55,202 DEBUG [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e2db600972e34421ab3a9ebd4e3f8ba0 2023-10-27 23:52:55,202 INFO [StoreOpener-9aca977d0db042f340c741ee9446ec83-1] regionserver.HStore(310): Store=9aca977d0db042f340c741ee9446ec83/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:55,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:55,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:55,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:52:55,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:55,210 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 9aca977d0db042f340c741ee9446ec83; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59358265, jitterRate=-0.11549292504787445}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:52:55,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 9aca977d0db042f340c741ee9446ec83: 2023-10-27 23:52:55,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83., pid=18, masterSystemTime=1698450774930 2023-10-27 23:52:55,211 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:52:55,212 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:52:55,212 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:52:55,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:52:55,212 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83. 2023-10-27 23:52:55,212 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83. 2023-10-27 23:52:55,212 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1912): 9aca977d0db042f340c741ee9446ec83/cf2 is initiating minor compaction (all files) 2023-10-27 23:52:55,212 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1912): 9aca977d0db042f340c741ee9446ec83/cf1 is initiating minor compaction (all files) 2023-10-27 23:52:55,212 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9aca977d0db042f340c741ee9446ec83/cf2 in testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83. 2023-10-27 23:52:55,212 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9aca977d0db042f340c741ee9446ec83/cf1 in testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83. 2023-10-27 23:52:55,213 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/68b4536a221a42628ce69ded1e41e018.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/68b4536a221a42628ce69ded1e41e018-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/5d07fa5f8b3d4b33818f9c523db84071.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/5d07fa5f8b3d4b33818f9c523db84071-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/a88b5a60a38644eaa0012136ec701fc0.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/a88b5a60a38644eaa0012136ec701fc0-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/10ad920c9db24c04afaf6e5823d8604a.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/10ad920c9db24c04afaf6e5823d8604a-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-9a6eb09cb9524fe7943e1ce637ff1842, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e8f365c77ec64b85ac82f587706cc82e, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-aaa71348029d4cc993f752bafef54c2e] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/.tmp, totalSize=34.9 K 2023-10-27 23:52:55,213 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/dd03d244abfe47159e74a09813bcaec7.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/dd03d244abfe47159e74a09813bcaec7-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/04ba4141a0f34d8eb85680b2928c30b8.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/04ba4141a0f34d8eb85680b2928c30b8-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/16dc81cf0aa24a738be12a15678ccb7f.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/16dc81cf0aa24a738be12a15678ccb7f-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/f3d88e7ea44640e098237e6df13ca0e0.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/f3d88e7ea44640e098237e6df13ca0e0-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-b117b97848c047f18901f0277d2b9494, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-53c3985a23aa4a0cbbe83ea9449370ec, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e2db600972e34421ab3a9ebd4e3f8ba0] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/.tmp, totalSize=34.9 K 2023-10-27 23:52:55,214 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 68b4536a221a42628ce69ded1e41e018.1a178e5e9b9e5a90d2396ab013c5750a, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1698450771066 2023-10-27 23:52:55,214 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting dd03d244abfe47159e74a09813bcaec7.1a178e5e9b9e5a90d2396ab013c5750a, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1698450771066 2023-10-27 23:52:55,218 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 5d07fa5f8b3d4b33818f9c523db84071.1a178e5e9b9e5a90d2396ab013c5750a, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1698450771513 2023-10-27 23:52:55,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83. 2023-10-27 23:52:55,218 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting 04ba4141a0f34d8eb85680b2928c30b8.1a178e5e9b9e5a90d2396ab013c5750a, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1698450771513 2023-10-27 23:52:55,218 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83. 2023-10-27 23:52:55,218 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=9aca977d0db042f340c741ee9446ec83, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:55,218 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1698450775218"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450775218"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450775218"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450775218"}]},"ts":"1698450775218"} 2023-10-27 23:52:55,219 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting a88b5a60a38644eaa0012136ec701fc0.1a178e5e9b9e5a90d2396ab013c5750a, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1698450771842 2023-10-27 23:52:55,219 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting 16dc81cf0aa24a738be12a15678ccb7f.1a178e5e9b9e5a90d2396ab013c5750a, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1698450771842 2023-10-27 23:52:55,221 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 10ad920c9db24c04afaf6e5823d8604a.1a178e5e9b9e5a90d2396ab013c5750a, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1698450772158 2023-10-27 23:52:55,221 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting f3d88e7ea44640e098237e6df13ca0e0.1a178e5e9b9e5a90d2396ab013c5750a, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1698450772158 2023-10-27 23:52:55,222 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-9a6eb09cb9524fe7943e1ce637ff1842, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450772508 2023-10-27 23:52:55,222 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-b117b97848c047f18901f0277d2b9494, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450772508 2023-10-27 23:52:55,223 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-53c3985a23aa4a0cbbe83ea9449370ec, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450772820 2023-10-27 23:52:55,223 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e8f365c77ec64b85ac82f587706cc82e, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450772820 2023-10-27 23:52:55,224 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e2db600972e34421ab3a9ebd4e3f8ba0, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1698450773615 2023-10-27 23:52:55,225 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-aaa71348029d4cc993f752bafef54c2e, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1698450773615 2023-10-27 23:52:55,227 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=18, resume processing ppid=16 2023-10-27 23:52:55,227 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=18, ppid=16, state=SUCCESS; OpenRegionProcedure 9aca977d0db042f340c741ee9446ec83, server=jenkins-hbase16.apache.org,36599,1698450767241 in 437 msec 2023-10-27 23:52:55,232 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=16, resume processing ppid=12 2023-10-27 23:52:55,232 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=16, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=9aca977d0db042f340c741ee9446ec83, ASSIGN in 608 msec 2023-10-27 23:52:55,236 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 in 1.5940 sec 2023-10-27 23:52:55,261 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] throttle.PressureAwareThroughputController(145): 9aca977d0db042f340c741ee9446ec83#cf2#compaction#17 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:52:55,261 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9aca977d0db042f340c741ee9446ec83#cf1#compaction#16 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:52:55,283 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/.tmp/cf1/4f2be4f388514d56a375e97188da8b4e as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/4f2be4f388514d56a375e97188da8b4e 2023-10-27 23:52:55,283 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/.tmp/cf2/8801f41450294b0b97575a75da4b24e4 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/8801f41450294b0b97575a75da4b24e4 2023-10-27 23:52:55,291 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9aca977d0db042f340c741ee9446ec83/cf2 of 9aca977d0db042f340c741ee9446ec83 into 8801f41450294b0b97575a75da4b24e4(size=6.7 K), total size for store is 6.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:52:55,291 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9aca977d0db042f340c741ee9446ec83/cf1 of 9aca977d0db042f340c741ee9446ec83 into 4f2be4f388514d56a375e97188da8b4e(size=6.7 K), total size for store is 6.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:52:55,291 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9aca977d0db042f340c741ee9446ec83: 2023-10-27 23:52:55,292 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9aca977d0db042f340c741ee9446ec83: 2023-10-27 23:52:55,292 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83., storeName=9aca977d0db042f340c741ee9446ec83/cf2, priority=9, startTime=1698450775212; duration=0sec 2023-10-27 23:52:55,292 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83., storeName=9aca977d0db042f340c741ee9446ec83/cf1, priority=9, startTime=1698450775211; duration=0sec 2023-10-27 23:52:55,292 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:52:55,292 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:52:55,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36599] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Scan size: 157 connection: 188.40.62.62:49794 deadline: 1698450835399, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a. is not online on jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:55,530 DEBUG [Listener at localhost/43373] client.ClientScanner(240): Advancing internal scanner to startKey at '26', inclusive 2023-10-27 23:52:55,542 INFO [Listener at localhost/43373] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-10-27 23:52:55,545 DEBUG [Listener at localhost/43373] client.ClientScanner(240): Advancing internal scanner to startKey at '26', inclusive 2023-10-27 23:52:55,574 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:52:55,584 INFO [Listener at localhost/43373] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionUnevenDaughter 2023-10-27 23:52:55,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable testSplitTableRegionUnevenDaughter 2023-10-27 23:52:55,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=19, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:55,602 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450775602"}]},"ts":"1698450775602"} 2023-10-27 23:52:55,604 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=DISABLING in hbase:meta 2023-10-27 23:52:55,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=19 2023-10-27 23:52:55,636 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionUnevenDaughter to state=DISABLING 2023-10-27 23:52:55,638 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=8285b45cae27573bc22fb00c23a6d6fe, UNASSIGN}, {pid=21, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=9aca977d0db042f340c741ee9446ec83, UNASSIGN}] 2023-10-27 23:52:55,640 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=9aca977d0db042f340c741ee9446ec83, UNASSIGN 2023-10-27 23:52:55,641 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9aca977d0db042f340c741ee9446ec83, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:55,641 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1698450775641"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450775641"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450775641"}]},"ts":"1698450775641"} 2023-10-27 23:52:55,643 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 9aca977d0db042f340c741ee9446ec83, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:52:55,647 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=8285b45cae27573bc22fb00c23a6d6fe, UNASSIGN 2023-10-27 23:52:55,648 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=8285b45cae27573bc22fb00c23a6d6fe, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:55,648 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1698450775648"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450775648"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450775648"}]},"ts":"1698450775648"} 2023-10-27 23:52:55,650 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=23, ppid=20, state=RUNNABLE; CloseRegionProcedure 8285b45cae27573bc22fb00c23a6d6fe, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:52:55,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=19 2023-10-27 23:52:55,796 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:55,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 8285b45cae27573bc22fb00c23a6d6fe 2023-10-27 23:52:55,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 8285b45cae27573bc22fb00c23a6d6fe, disabling compactions & flushes 2023-10-27 23:52:55,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe. 2023-10-27 23:52:55,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe. 2023-10-27 23:52:55,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe. after waiting 0 ms 2023-10-27 23:52:55,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe. 2023-10-27 23:52:55,802 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/68b4536a221a42628ce69ded1e41e018.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/68b4536a221a42628ce69ded1e41e018-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/5d07fa5f8b3d4b33818f9c523db84071.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/5d07fa5f8b3d4b33818f9c523db84071-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/a88b5a60a38644eaa0012136ec701fc0.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/a88b5a60a38644eaa0012136ec701fc0-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/10ad920c9db24c04afaf6e5823d8604a.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/10ad920c9db24c04afaf6e5823d8604a-bottom] to archive 2023-10-27 23:52:55,803 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:52:55,809 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/68b4536a221a42628ce69ded1e41e018.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/68b4536a221a42628ce69ded1e41e018.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,810 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/5d07fa5f8b3d4b33818f9c523db84071.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/5d07fa5f8b3d4b33818f9c523db84071.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,812 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/a88b5a60a38644eaa0012136ec701fc0.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/a88b5a60a38644eaa0012136ec701fc0.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,813 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/10ad920c9db24c04afaf6e5823d8604a.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/10ad920c9db24c04afaf6e5823d8604a.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,838 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/dd03d244abfe47159e74a09813bcaec7.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/dd03d244abfe47159e74a09813bcaec7-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/04ba4141a0f34d8eb85680b2928c30b8.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/04ba4141a0f34d8eb85680b2928c30b8-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/16dc81cf0aa24a738be12a15678ccb7f.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/16dc81cf0aa24a738be12a15678ccb7f-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/f3d88e7ea44640e098237e6df13ca0e0.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/f3d88e7ea44640e098237e6df13ca0e0-bottom] to archive 2023-10-27 23:52:55,840 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:52:55,842 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/dd03d244abfe47159e74a09813bcaec7.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/dd03d244abfe47159e74a09813bcaec7.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,844 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/04ba4141a0f34d8eb85680b2928c30b8.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/04ba4141a0f34d8eb85680b2928c30b8.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,846 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/16dc81cf0aa24a738be12a15678ccb7f.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/16dc81cf0aa24a738be12a15678ccb7f.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,848 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/f3d88e7ea44640e098237e6df13ca0e0.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/f3d88e7ea44640e098237e6df13ca0e0.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-10-27 23:52:55,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:52:55,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe. 2023-10-27 23:52:55,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 8285b45cae27573bc22fb00c23a6d6fe: 2023-10-27 23:52:55,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 8285b45cae27573bc22fb00c23a6d6fe 2023-10-27 23:52:55,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:55,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 9aca977d0db042f340c741ee9446ec83, disabling compactions & flushes 2023-10-27 23:52:55,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83. 2023-10-27 23:52:55,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83. 2023-10-27 23:52:55,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83. after waiting 0 ms 2023-10-27 23:52:55,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83. 2023-10-27 23:52:55,858 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=8285b45cae27573bc22fb00c23a6d6fe, regionState=CLOSED 2023-10-27 23:52:55,859 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1698450775858"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450775858"}]},"ts":"1698450775858"} 2023-10-27 23:52:55,860 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/68b4536a221a42628ce69ded1e41e018.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/68b4536a221a42628ce69ded1e41e018-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/5d07fa5f8b3d4b33818f9c523db84071.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/5d07fa5f8b3d4b33818f9c523db84071-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/a88b5a60a38644eaa0012136ec701fc0.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/a88b5a60a38644eaa0012136ec701fc0-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/10ad920c9db24c04afaf6e5823d8604a.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/10ad920c9db24c04afaf6e5823d8604a-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-9a6eb09cb9524fe7943e1ce637ff1842, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e8f365c77ec64b85ac82f587706cc82e, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-aaa71348029d4cc993f752bafef54c2e] to archive 2023-10-27 23:52:55,862 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:52:55,864 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/68b4536a221a42628ce69ded1e41e018.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/68b4536a221a42628ce69ded1e41e018.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,866 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/5d07fa5f8b3d4b33818f9c523db84071.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/5d07fa5f8b3d4b33818f9c523db84071.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,867 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=23, resume processing ppid=20 2023-10-27 23:52:55,868 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/a88b5a60a38644eaa0012136ec701fc0.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/a88b5a60a38644eaa0012136ec701fc0.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,876 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=23, ppid=20, state=SUCCESS; CloseRegionProcedure 8285b45cae27573bc22fb00c23a6d6fe, server=jenkins-hbase16.apache.org,36599,1698450767241 in 213 msec 2023-10-27 23:52:55,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=20, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=8285b45cae27573bc22fb00c23a6d6fe, UNASSIGN in 239 msec 2023-10-27 23:52:55,879 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/10ad920c9db24c04afaf6e5823d8604a.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/10ad920c9db24c04afaf6e5823d8604a.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,881 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-9a6eb09cb9524fe7943e1ce637ff1842 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-9a6eb09cb9524fe7943e1ce637ff1842 2023-10-27 23:52:55,883 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e8f365c77ec64b85ac82f587706cc82e to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e8f365c77ec64b85ac82f587706cc82e 2023-10-27 23:52:55,885 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-aaa71348029d4cc993f752bafef54c2e to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-aaa71348029d4cc993f752bafef54c2e 2023-10-27 23:52:55,888 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/dd03d244abfe47159e74a09813bcaec7.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/dd03d244abfe47159e74a09813bcaec7-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/04ba4141a0f34d8eb85680b2928c30b8.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/04ba4141a0f34d8eb85680b2928c30b8-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/16dc81cf0aa24a738be12a15678ccb7f.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/16dc81cf0aa24a738be12a15678ccb7f-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/f3d88e7ea44640e098237e6df13ca0e0.1a178e5e9b9e5a90d2396ab013c5750a->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/f3d88e7ea44640e098237e6df13ca0e0-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-b117b97848c047f18901f0277d2b9494, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-53c3985a23aa4a0cbbe83ea9449370ec, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e2db600972e34421ab3a9ebd4e3f8ba0] to archive 2023-10-27 23:52:55,889 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:52:55,892 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/dd03d244abfe47159e74a09813bcaec7.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/dd03d244abfe47159e74a09813bcaec7.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,894 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/04ba4141a0f34d8eb85680b2928c30b8.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/04ba4141a0f34d8eb85680b2928c30b8.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,896 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/16dc81cf0aa24a738be12a15678ccb7f.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/16dc81cf0aa24a738be12a15678ccb7f.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,898 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/f3d88e7ea44640e098237e6df13ca0e0.1a178e5e9b9e5a90d2396ab013c5750a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/f3d88e7ea44640e098237e6df13ca0e0.1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:55,900 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-b117b97848c047f18901f0277d2b9494 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-b117b97848c047f18901f0277d2b9494 2023-10-27 23:52:55,901 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-53c3985a23aa4a0cbbe83ea9449370ec to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-53c3985a23aa4a0cbbe83ea9449370ec 2023-10-27 23:52:55,903 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e2db600972e34421ab3a9ebd4e3f8ba0 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/testSplitTableRegionUnevenDaughter=1a178e5e9b9e5a90d2396ab013c5750a-e2db600972e34421ab3a9ebd4e3f8ba0 2023-10-27 23:52:55,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-10-27 23:52:55,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:52:55,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83. 2023-10-27 23:52:55,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 9aca977d0db042f340c741ee9446ec83: 2023-10-27 23:52:55,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:55,914 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9aca977d0db042f340c741ee9446ec83, regionState=CLOSED 2023-10-27 23:52:55,914 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1698450775913"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450775913"}]},"ts":"1698450775913"} 2023-10-27 23:52:55,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=19 2023-10-27 23:52:55,919 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=22, resume processing ppid=21 2023-10-27 23:52:55,919 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 9aca977d0db042f340c741ee9446ec83, server=jenkins-hbase16.apache.org,36599,1698450767241 in 273 msec 2023-10-27 23:52:55,923 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=21, resume processing ppid=19 2023-10-27 23:52:55,923 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=21, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=9aca977d0db042f340c741ee9446ec83, UNASSIGN in 281 msec 2023-10-27 23:52:55,925 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450775924"}]},"ts":"1698450775924"} 2023-10-27 23:52:55,927 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=DISABLED in hbase:meta 2023-10-27 23:52:55,958 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionUnevenDaughter to state=DISABLED 2023-10-27 23:52:55,961 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter in 369 msec 2023-10-27 23:52:56,188 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-10-27 23:52:56,188 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-10-27 23:52:56,189 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:52:56,189 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro Metrics about HBase RegionObservers 2023-10-27 23:52:56,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=19 2023-10-27 23:52:56,217 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionUnevenDaughter, procId: 19 completed 2023-10-27 23:52:56,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete testSplitTableRegionUnevenDaughter 2023-10-27 23:52:56,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=24, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:56,230 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=24, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:56,233 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=24, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:56,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=24 2023-10-27 23:52:56,241 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:56,241 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe 2023-10-27 23:52:56,241 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:56,244 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/recovered.edits] 2023-10-27 23:52:56,244 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/recovered.edits] 2023-10-27 23:52:56,244 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/recovered.edits] 2023-10-27 23:52:56,250 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/10ad920c9db24c04afaf6e5823d8604a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/10ad920c9db24c04afaf6e5823d8604a 2023-10-27 23:52:56,250 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/4f2be4f388514d56a375e97188da8b4e to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf1/4f2be4f388514d56a375e97188da8b4e 2023-10-27 23:52:56,250 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/a57b39a149744799888f8b8b817a11ec to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf1/a57b39a149744799888f8b8b817a11ec 2023-10-27 23:52:56,252 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/5d07fa5f8b3d4b33818f9c523db84071 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/5d07fa5f8b3d4b33818f9c523db84071 2023-10-27 23:52:56,253 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/68b4536a221a42628ce69ded1e41e018 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/68b4536a221a42628ce69ded1e41e018 2023-10-27 23:52:56,253 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/8801f41450294b0b97575a75da4b24e4 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/cf2/8801f41450294b0b97575a75da4b24e4 2023-10-27 23:52:56,254 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/068c1f976790455bb5f77d7e1a552f36 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/cf2/068c1f976790455bb5f77d7e1a552f36 2023-10-27 23:52:56,255 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/9a6eb09cb9524fe7943e1ce637ff1842 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/9a6eb09cb9524fe7943e1ce637ff1842 2023-10-27 23:52:56,257 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/a88b5a60a38644eaa0012136ec701fc0 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/a88b5a60a38644eaa0012136ec701fc0 2023-10-27 23:52:56,258 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/recovered.edits/90.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe/recovered.edits/90.seqid 2023-10-27 23:52:56,258 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/recovered.edits/90.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83/recovered.edits/90.seqid 2023-10-27 23:52:56,259 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:56,260 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/8285b45cae27573bc22fb00c23a6d6fe 2023-10-27 23:52:56,260 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/aaa71348029d4cc993f752bafef54c2e to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/aaa71348029d4cc993f752bafef54c2e 2023-10-27 23:52:56,261 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/e8f365c77ec64b85ac82f587706cc82e to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf1/e8f365c77ec64b85ac82f587706cc82e 2023-10-27 23:52:56,264 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/04ba4141a0f34d8eb85680b2928c30b8 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/04ba4141a0f34d8eb85680b2928c30b8 2023-10-27 23:52:56,266 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/16dc81cf0aa24a738be12a15678ccb7f to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/16dc81cf0aa24a738be12a15678ccb7f 2023-10-27 23:52:56,268 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/53c3985a23aa4a0cbbe83ea9449370ec to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/53c3985a23aa4a0cbbe83ea9449370ec 2023-10-27 23:52:56,270 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/b117b97848c047f18901f0277d2b9494 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/b117b97848c047f18901f0277d2b9494 2023-10-27 23:52:56,272 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/dd03d244abfe47159e74a09813bcaec7 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/dd03d244abfe47159e74a09813bcaec7 2023-10-27 23:52:56,273 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/e2db600972e34421ab3a9ebd4e3f8ba0 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/e2db600972e34421ab3a9ebd4e3f8ba0 2023-10-27 23:52:56,275 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/f3d88e7ea44640e098237e6df13ca0e0 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/cf2/f3d88e7ea44640e098237e6df13ca0e0 2023-10-27 23:52:56,279 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/recovered.edits/85.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a/recovered.edits/85.seqid 2023-10-27 23:52:56,280 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionUnevenDaughter/1a178e5e9b9e5a90d2396ab013c5750a 2023-10-27 23:52:56,280 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionUnevenDaughter regions 2023-10-27 23:52:56,283 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=24, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:56,307 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionUnevenDaughter from hbase:meta 2023-10-27 23:52:56,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=24 2023-10-27 23:52:56,347 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionUnevenDaughter' descriptor. 2023-10-27 23:52:56,349 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=24, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:56,349 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionUnevenDaughter' from region states. 2023-10-27 23:52:56,350 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450776350"}]},"ts":"9223372036854775807"} 2023-10-27 23:52:56,350 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450776350"}]},"ts":"9223372036854775807"} 2023-10-27 23:52:56,350 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450776350"}]},"ts":"9223372036854775807"} 2023-10-27 23:52:56,353 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-10-27 23:52:56,353 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1a178e5e9b9e5a90d2396ab013c5750a, NAME => 'testSplitTableRegionUnevenDaughter,,1698450770308.1a178e5e9b9e5a90d2396ab013c5750a.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 8285b45cae27573bc22fb00c23a6d6fe, NAME => 'testSplitTableRegionUnevenDaughter,,1698450773638.8285b45cae27573bc22fb00c23a6d6fe.', STARTKEY => '', ENDKEY => '26'}, {ENCODED => 9aca977d0db042f340c741ee9446ec83, NAME => 'testSplitTableRegionUnevenDaughter,26,1698450773638.9aca977d0db042f340c741ee9446ec83.', STARTKEY => '26', ENDKEY => ''}] 2023-10-27 23:52:56,353 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionUnevenDaughter' as deleted. 2023-10-27 23:52:56,354 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1698450776354"}]},"ts":"9223372036854775807"} 2023-10-27 23:52:56,356 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionUnevenDaughter state from META 2023-10-27 23:52:56,393 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=24, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:56,394 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter in 170 msec 2023-10-27 23:52:56,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=24 2023-10-27 23:52:56,538 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionUnevenDaughter, procId: 24 completed 2023-10-27 23:52:56,553 INFO [Listener at localhost/43373] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionUnevenDaughter Thread=392 (was 361) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:55048 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-metaLookup-shared--pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x27f582be-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:37152 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:43714 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x27f582be-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:43512 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase16:36599-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:43558 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:57432 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:57494 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:54816 [Waiting for operation #21] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:37166 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:45620 [Waiting for operation #20] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:55032 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:43548 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:55076 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x27f582be-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x27f582be-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:43698 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:57518 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x27f582be-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:55046 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=872 (was 698) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=400 (was 392) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=5732 (was 5827) 2023-10-27 23:52:56,566 INFO [Listener at localhost/43373] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitWithoutPONR Thread=392, OpenFileDescriptor=872, MaxFileDescriptor=60000, SystemLoadAverage=400, ProcessCount=170, AvailableMemoryMB=5732 2023-10-27 23:52:56,567 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:52:56,601 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-10-27 23:52:56,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(492): Client=jenkins//188.40.62.62 set balanceSwitch=false 2023-10-27 23:52:56,603 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=25, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:52:56,605 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_PRE_OPERATION 2023-10-27 23:52:56,606 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-10-27 23:52:56,608 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:52:56,609 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab empty. 2023-10-27 23:52:56,609 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:52:56,609 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitWithoutPONR regions 2023-10-27 23:52:56,623 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/.tabledesc/.tableinfo.0000000001 2023-10-27 23:52:56,625 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(7675): creating {ENCODED => a6d7da66a8a7b852f46fc7477e45d6ab, NAME => 'testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitWithoutPONR', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp 2023-10-27 23:52:56,637 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(866): Instantiated testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:52:56,637 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1604): Closing a6d7da66a8a7b852f46fc7477e45d6ab, disabling compactions & flushes 2023-10-27 23:52:56,637 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1626): Closing region testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:56,637 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:56,637 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. after waiting 0 ms 2023-10-27 23:52:56,637 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:56,637 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1838): Closed testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:56,637 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1558): Region close journal for a6d7da66a8a7b852f46fc7477e45d6ab: 2023-10-27 23:52:56,640 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_ADD_TO_META 2023-10-27 23:52:56,641 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450776641"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450776641"}]},"ts":"1698450776641"} 2023-10-27 23:52:56,643 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-10-27 23:52:56,644 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-10-27 23:52:56,645 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450776644"}]},"ts":"1698450776644"} 2023-10-27 23:52:56,646 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=ENABLING in hbase:meta 2023-10-27 23:52:56,666 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-10-27 23:52:56,668 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-27 23:52:56,668 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-10-27 23:52:56,668 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-10-27 23:52:56,668 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-10-27 23:52:56,668 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, ASSIGN}] 2023-10-27 23:52:56,671 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, ASSIGN 2023-10-27 23:52:56,672 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,36599,1698450767241; forceNewPlan=false, retain=false 2023-10-27 23:52:56,822 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-10-27 23:52:56,823 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:56,823 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450776823"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450776823"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450776823"}]},"ts":"1698450776823"} 2023-10-27 23:52:56,826 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; OpenRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:52:56,978 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:56,981 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:56,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a6d7da66a8a7b852f46fc7477e45d6ab, NAME => 'testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:52:56,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:52:56,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:52:56,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:52:56,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:52:56,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:52:56,984 INFO [StoreOpener-a6d7da66a8a7b852f46fc7477e45d6ab-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:52:56,985 DEBUG [StoreOpener-a6d7da66a8a7b852f46fc7477e45d6ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1 2023-10-27 23:52:56,985 DEBUG [StoreOpener-a6d7da66a8a7b852f46fc7477e45d6ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1 2023-10-27 23:52:56,986 INFO [StoreOpener-a6d7da66a8a7b852f46fc7477e45d6ab-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a6d7da66a8a7b852f46fc7477e45d6ab columnFamilyName cf1 2023-10-27 23:52:56,987 INFO [StoreOpener-a6d7da66a8a7b852f46fc7477e45d6ab-1] regionserver.HStore(310): Store=a6d7da66a8a7b852f46fc7477e45d6ab/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:56,987 INFO [StoreOpener-a6d7da66a8a7b852f46fc7477e45d6ab-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:52:56,989 DEBUG [StoreOpener-a6d7da66a8a7b852f46fc7477e45d6ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2 2023-10-27 23:52:56,989 DEBUG [StoreOpener-a6d7da66a8a7b852f46fc7477e45d6ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2 2023-10-27 23:52:56,990 INFO [StoreOpener-a6d7da66a8a7b852f46fc7477e45d6ab-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a6d7da66a8a7b852f46fc7477e45d6ab columnFamilyName cf2 2023-10-27 23:52:56,991 INFO [StoreOpener-a6d7da66a8a7b852f46fc7477e45d6ab-1] regionserver.HStore(310): Store=a6d7da66a8a7b852f46fc7477e45d6ab/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:52:56,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:52:56,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:52:56,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:52:56,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:52:56,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-10-27 23:52:56,999 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened a6d7da66a8a7b852f46fc7477e45d6ab; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59122553, jitterRate=-0.11900530755519867}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:52:56,999 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for a6d7da66a8a7b852f46fc7477e45d6ab: 2023-10-27 23:52:57,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab., pid=27, masterSystemTime=1698450776978 2023-10-27 23:52:57,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:57,002 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:57,003 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,003 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450777003"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450777003"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450777003"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450777003"}]},"ts":"1698450777003"} 2023-10-27 23:52:57,008 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=27, resume processing ppid=26 2023-10-27 23:52:57,008 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=27, ppid=26, state=SUCCESS; OpenRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 in 180 msec 2023-10-27 23:52:57,010 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=26, resume processing ppid=25 2023-10-27 23:52:57,010 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=26, ppid=25, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, ASSIGN in 340 msec 2023-10-27 23:52:57,011 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-10-27 23:52:57,011 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450777011"}]},"ts":"1698450777011"} 2023-10-27 23:52:57,013 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=ENABLED in hbase:meta 2023-10-27 23:52:57,021 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_POST_OPERATION 2023-10-27 23:52:57,022 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR in 419 msec 2023-10-27 23:52:57,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:52:57,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-10-27 23:52:57,117 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-10-27 23:52:57,117 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,118 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-10-27 23:52:57,118 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-10-27 23:52:57,119 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,119 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,128 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,128 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,128 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,128 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,128 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,129 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:57,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:57,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:57,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:57,130 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:57,130 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:57,130 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,130 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:57,133 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:57,134 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:57,134 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,133 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:57,134 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:57,134 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:57,134 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:57,134 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:57,134 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:57,134 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:52:57,134 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:52:57,135 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:57,135 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. started... 2023-10-27 23:52:57,135 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6d7da66a8a7b852f46fc7477e45d6ab 2/2 column families, dataSize=120 B heapSize=928 B 2023-10-27 23:52:57,144 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,147 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/4609a3ebc1eb4c9283d184a3e62f39e9 2023-10-27 23:52:57,153 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,153 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,153 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:57,153 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,153 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:57,161 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/aefddd22bc754dc5b3eca560340c2f8b 2023-10-27 23:52:57,167 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/4609a3ebc1eb4c9283d184a3e62f39e9 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9 2023-10-27 23:52:57,173 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:52:57,174 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/aefddd22bc754dc5b3eca560340c2f8b as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b 2023-10-27 23:52:57,179 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:52:57,180 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for a6d7da66a8a7b852f46fc7477e45d6ab in 45ms, sequenceid=6, compaction requested=false 2023-10-27 23:52:57,180 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitWithoutPONR' 2023-10-27 23:52:57,181 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6d7da66a8a7b852f46fc7477e45d6ab: 2023-10-27 23:52:57,181 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:57,181 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:52:57,181 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:52:57,181 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,181 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:57,181 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:57,191 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,191 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:57,192 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,192 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:57,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-10-27 23:52:57,194 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-10-27 23:52:57,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47621feb[Count = 0] remaining members to acquire global barrier 2023-10-27 23:52:57,194 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,203 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,203 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,203 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,203 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,203 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:52:57,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,203 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:57,203 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,203 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:52:57,204 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:57,204 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,204 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:52:57,204 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:57,204 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:57,204 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:52:57,211 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:57,211 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,211 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:57,211 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:57,219 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:57,220 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:57,220 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:57,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:57,228 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:57,228 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:57,228 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase16.apache.org,36599,1698450767241': 2023-10-27 23:52:57,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-10-27 23:52:57,233 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-10-27 23:52:57,234 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:52:57,234 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:52:57,234 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-10-27 23:52:57,234 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:52:57,244 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,244 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,244 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,244 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,245 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:57,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,245 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,245 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,245 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:57,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,246 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,246 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:57,248 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,248 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,248 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-10-27 23:52:57,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:52:57,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,258 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:57,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,259 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-10-27 23:52:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,259 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:52:57,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,260 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,359 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:52:57,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:52:57,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:52:57,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-10-27 23:52:57,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-10-27 23:52:57,383 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-10-27 23:52:57,384 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,384 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-10-27 23:52:57,384 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-10-27 23:52:57,385 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,385 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,395 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,395 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,395 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,395 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,395 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,395 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:52:57,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:57,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:57,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:57,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:57,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:57,396 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:57,397 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:57,397 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,397 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,397 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:57,398 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:57,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:57,398 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:57,398 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:57,398 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:52:57,398 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:52:57,398 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:57,398 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:57,398 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,398 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. started... 2023-10-27 23:52:57,399 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:57,399 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:57,399 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:57,399 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6d7da66a8a7b852f46fc7477e45d6ab 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:52:57,408 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,409 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/62d169ea60984148b3a4bf88cfa16795 2023-10-27 23:52:57,416 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,416 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:57,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:57,425 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/de202d94265f41549c744b6d939d1fb5 2023-10-27 23:52:57,432 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/62d169ea60984148b3a4bf88cfa16795 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795 2023-10-27 23:52:57,437 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:52:57,438 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/de202d94265f41549c744b6d939d1fb5 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5 2023-10-27 23:52:57,444 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:52:57,445 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a6d7da66a8a7b852f46fc7477e45d6ab in 46ms, sequenceid=19, compaction requested=false 2023-10-27 23:52:57,445 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6d7da66a8a7b852f46fc7477e45d6ab: 2023-10-27 23:52:57,445 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:57,445 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:52:57,445 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:52:57,445 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,445 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:57,445 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:57,453 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,453 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:57,453 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,453 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:57,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-10-27 23:52:57,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@523e6af2[Count = 0] remaining members to acquire global barrier 2023-10-27 23:52:57,455 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-10-27 23:52:57,455 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,461 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,461 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,461 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,462 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:57,462 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:57,462 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:57,461 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,462 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:52:57,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,462 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:52:57,462 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,462 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:52:57,462 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:57,462 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:57,462 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:52:57,485 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:57,485 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:57,485 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:57,494 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:57,494 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,494 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:57,495 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:57,495 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:57,495 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:57,495 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase16.apache.org,36599,1698450767241': 2023-10-27 23:52:57,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-10-27 23:52:57,498 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-10-27 23:52:57,498 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:52:57,498 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:52:57,498 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-10-27 23:52:57,498 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:52:57,508 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,508 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,508 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,508 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,508 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,508 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,508 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,509 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:57,509 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:57,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,509 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,511 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,512 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:57,512 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-10-27 23:52:57,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:52:57,520 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:57,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,521 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-10-27 23:52:57,521 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:52:57,521 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,521 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,521 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,521 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,521 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,521 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,621 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:52:57,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:52:57,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:52:57,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-10-27 23:52:57,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-10-27 23:52:57,647 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-10-27 23:52:57,647 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,648 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-10-27 23:52:57,648 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-10-27 23:52:57,648 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,649 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,658 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,658 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,658 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,658 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,658 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,659 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:57,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:57,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:57,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:57,660 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:57,660 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:57,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:57,660 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:57,660 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:57,661 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:57,661 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:57,661 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:57,661 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:57,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:57,661 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:52:57,661 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:57,661 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:52:57,661 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:57,661 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. started... 2023-10-27 23:52:57,661 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,662 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:57,662 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:57,662 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:57,662 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6d7da66a8a7b852f46fc7477e45d6ab 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:52:57,669 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,673 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/b0c83c29a88d4cfb872b9cd9720c6190 2023-10-27 23:52:57,678 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,678 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:57,678 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,678 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,678 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:57,688 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/c9a2e59aa69e4f758834d00a2f4796a1 2023-10-27 23:52:57,694 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/b0c83c29a88d4cfb872b9cd9720c6190 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190 2023-10-27 23:52:57,700 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:52:57,701 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/c9a2e59aa69e4f758834d00a2f4796a1 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1 2023-10-27 23:52:57,706 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:52:57,707 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a6d7da66a8a7b852f46fc7477e45d6ab in 45ms, sequenceid=32, compaction requested=true 2023-10-27 23:52:57,707 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6d7da66a8a7b852f46fc7477e45d6ab: 2023-10-27 23:52:57,708 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:57,708 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:52:57,708 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:52:57,708 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,708 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:57,708 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:57,716 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,716 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:57,717 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,717 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:57,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-10-27 23:52:57,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28d0d7e3[Count = 0] remaining members to acquire global barrier 2023-10-27 23:52:57,718 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-10-27 23:52:57,718 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,728 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,728 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,728 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,728 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:57,728 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:57,728 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:52:57,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,728 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,729 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:52:57,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,729 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:57,729 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:57,729 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:57,729 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:57,729 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:52:57,729 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:52:57,736 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:57,736 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:57,736 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:57,745 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:57,745 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:57,745 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:57,745 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,745 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:57,745 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:57,745 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase16.apache.org,36599,1698450767241': 2023-10-27 23:52:57,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-10-27 23:52:57,748 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-10-27 23:52:57,748 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:52:57,748 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:52:57,748 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-10-27 23:52:57,748 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:52:57,758 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,758 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,758 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:57,758 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,758 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,758 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,758 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,758 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,759 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,759 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,761 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:57,761 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,762 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,770 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,770 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,770 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,770 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,770 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,770 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,770 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,770 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,770 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:57,770 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,770 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,770 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,770 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,771 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,771 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,771 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,771 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-10-27 23:52:57,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:52:57,770 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:57,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:57,771 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-10-27 23:52:57,771 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:52:57,872 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:52:57,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:52:57,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:52:57,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-10-27 23:52:57,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-10-27 23:52:57,897 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-10-27 23:52:57,897 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,898 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-10-27 23:52:57,898 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-10-27 23:52:57,898 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,898 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,923 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,923 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,923 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,923 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:57,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:57,924 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,924 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:52:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:57,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:57,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:57,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:57,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:57,925 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:57,926 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:57,926 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,926 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:57,927 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:57,927 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:57,927 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:57,927 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:57,929 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:52:57,929 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:57,929 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:52:57,929 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:57,929 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:57,929 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:57,929 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:57,929 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:57,929 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. started... 2023-10-27 23:52:57,930 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6d7da66a8a7b852f46fc7477e45d6ab 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:52:57,940 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/f2dfa3131f604f7588765386c7a0da7b 2023-10-27 23:52:57,941 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,949 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,949 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,949 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:57,950 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,950 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:57,955 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/1c24b0739cc246ff8abbb83025837bf7 2023-10-27 23:52:57,961 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/f2dfa3131f604f7588765386c7a0da7b as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b 2023-10-27 23:52:57,967 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:52:57,968 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/1c24b0739cc246ff8abbb83025837bf7 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7 2023-10-27 23:52:57,974 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:52:57,975 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a6d7da66a8a7b852f46fc7477e45d6ab in 44ms, sequenceid=45, compaction requested=true 2023-10-27 23:52:57,975 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6d7da66a8a7b852f46fc7477e45d6ab: 2023-10-27 23:52:57,975 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:57,975 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:52:57,975 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:52:57,975 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,975 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:57,975 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:57,983 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,983 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:57,983 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,983 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:57,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:57,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-10-27 23:52:57,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e1eaf65[Count = 0] remaining members to acquire global barrier 2023-10-27 23:52:57,985 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-10-27 23:52:57,985 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,995 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,995 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,995 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,995 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:57,995 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:57,995 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:52:57,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,995 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:57,995 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:57,995 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:57,995 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:52:57,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,995 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:52:57,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:57,996 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:57,996 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:57,996 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:52:58,003 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:58,003 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:58,003 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:58,011 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:58,011 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,011 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:58,011 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:58,011 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:58,012 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:58,012 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase16.apache.org,36599,1698450767241': 2023-10-27 23:52:58,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-10-27 23:52:58,015 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-10-27 23:52:58,015 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:52:58,015 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:52:58,015 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-10-27 23:52:58,015 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:52:58,024 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,024 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,024 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,024 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,025 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,025 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,025 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:58,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,025 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,026 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,026 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,027 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:58,028 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,028 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,036 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,036 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,036 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,036 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:58,036 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,036 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,037 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,036 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-10-27 23:52:58,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:52:58,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,036 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,037 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,037 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,037 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,037 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,037 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-10-27 23:52:58,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,037 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,038 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:52:58,038 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,038 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,038 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,038 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,138 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:52:58,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:52:58,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:52:58,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-10-27 23:52:58,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-10-27 23:52:58,166 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-10-27 23:52:58,166 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:58,166 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-10-27 23:52:58,166 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-10-27 23:52:58,167 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,167 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,174 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,174 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,174 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,174 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,175 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,175 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:52:58,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:58,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:58,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:58,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:58,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:58,176 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:58,176 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:58,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:58,176 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:58,176 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:58,176 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:58,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:58,176 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:58,176 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:58,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:58,177 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:52:58,177 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:58,176 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:58,177 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:58,177 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:58,177 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:52:58,177 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. started... 2023-10-27 23:52:58,177 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:58,177 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:58,177 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:58,177 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6d7da66a8a7b852f46fc7477e45d6ab 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:52:58,186 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,189 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/6bb4b637cac14a1788c8db687836756f 2023-10-27 23:52:58,194 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,194 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,194 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:58,195 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,195 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:58,204 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/5aa45f7e575a46fc8befa178336ab902 2023-10-27 23:52:58,209 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/6bb4b637cac14a1788c8db687836756f as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f 2023-10-27 23:52:58,214 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:52:58,214 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/5aa45f7e575a46fc8befa178336ab902 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902 2023-10-27 23:52:58,220 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:52:58,221 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a6d7da66a8a7b852f46fc7477e45d6ab in 44ms, sequenceid=58, compaction requested=true 2023-10-27 23:52:58,221 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6d7da66a8a7b852f46fc7477e45d6ab: 2023-10-27 23:52:58,221 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:58,221 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:52:58,221 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:52:58,221 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,222 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:58,222 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:58,228 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,228 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:58,228 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,228 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:58,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-10-27 23:52:58,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9d46e0a[Count = 0] remaining members to acquire global barrier 2023-10-27 23:52:58,230 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-10-27 23:52:58,230 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,236 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,236 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,236 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,237 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:58,237 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:58,237 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,237 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:52:58,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,237 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:52:58,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,237 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:58,237 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:58,237 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:58,237 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:52:58,237 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:58,237 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:52:58,245 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:58,245 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:58,245 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:58,253 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,253 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:58,253 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:58,253 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:58,261 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:58,261 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:58,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:58,261 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:58,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase16.apache.org,36599,1698450767241': 2023-10-27 23:52:58,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-10-27 23:52:58,265 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-10-27 23:52:58,265 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:52:58,265 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:52:58,265 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-10-27 23:52:58,265 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:52:58,291 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,291 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,291 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,291 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,291 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:58,291 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,292 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:58,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,292 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,292 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,293 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:58,294 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,294 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,294 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,308 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,308 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,308 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,308 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,308 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-10-27 23:52:58,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:52:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,308 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,308 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:58,308 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,308 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,308 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,309 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,309 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-10-27 23:52:58,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,309 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:52:58,309 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,309 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,309 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,309 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,309 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,309 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,309 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,409 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:52:58,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:52:58,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:52:58,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-10-27 23:52:58,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-10-27 23:52:58,437 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-10-27 23:52:58,437 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:58,437 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-10-27 23:52:58,437 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-10-27 23:52:58,437 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,437 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,444 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,445 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,445 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,444 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,445 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,445 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:52:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:58,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:58,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:52:58,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:58,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:58,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-10-27 23:52:58,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:58,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:58,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:58,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-10-27 23:52:58,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:58,447 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:58,447 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:58,447 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:58,447 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:58,447 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:58,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-10-27 23:52:58,447 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:58,447 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-10-27 23:52:58,447 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:58,447 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:52:58,447 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:58,447 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:58,447 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-10-27 23:52:58,448 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:58,448 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:52:58,448 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:52:58,448 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:58,448 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. started... 2023-10-27 23:52:58,448 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a6d7da66a8a7b852f46fc7477e45d6ab 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:52:58,466 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e 2023-10-27 23:52:58,469 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,478 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,478 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:58,478 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,478 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,478 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:58,489 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/51df109353864f6daa7599a3e1a1ed2b 2023-10-27 23:52:58,496 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e 2023-10-27 23:52:58,502 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:52:58,503 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/51df109353864f6daa7599a3e1a1ed2b as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b 2023-10-27 23:52:58,510 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:52:58,511 INFO [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a6d7da66a8a7b852f46fc7477e45d6ab in 63ms, sequenceid=71, compaction requested=true 2023-10-27 23:52:58,511 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a6d7da66a8a7b852f46fc7477e45d6ab: 2023-10-27 23:52:58,511 DEBUG [rs(jenkins-hbase16.apache.org,36599,1698450767241)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:52:58,511 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:52:58,511 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:52:58,511 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,511 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-10-27 23:52:58,511 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-10-27 23:52:58,519 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,519 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:58,520 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,520 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:52:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:58,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-10-27 23:52:58,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@289cec38[Count = 0] remaining members to acquire global barrier 2023-10-27 23:52:58,522 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-10-27 23:52:58,522 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,528 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,528 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,528 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,528 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:58,528 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:58,528 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:52:58,528 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:58,528 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:58,529 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:52:58,528 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,529 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:52:58,529 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-10-27 23:52:58,529 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-10-27 23:52:58,529 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:52:58,536 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:58,536 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,536 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:58,536 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:58,545 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:58,545 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:58,545 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:58,553 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-10-27 23:52:58,553 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:58,553 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-10-27 23:52:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase16.apache.org,36599,1698450767241': 2023-10-27 23:52:58,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-10-27 23:52:58,556 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-10-27 23:52:58,556 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:52:58,556 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:52:58,556 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-10-27 23:52:58,556 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:52:58,566 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,566 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,566 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,566 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:52:58,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,566 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,566 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,566 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,567 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:52:58,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,567 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:52:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,568 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:52:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,569 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,569 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:52:58,570 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-10-27 23:52:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,578 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,578 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,578 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-10-27 23:52:58,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:52:58,578 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:52:58,578 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,578 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,578 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:52:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,578 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-10-27 23:52:58,578 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,579 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,579 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:52:58,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:52:58,578 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:52:58,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:52:58,579 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,579 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-10-27 23:52:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,579 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:52:58,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:52:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:52:58,579 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,579 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,579 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:52:58,579 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-10-27 23:52:58,579 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-10-27 23:52:58,679 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:52:58,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:52:58,947 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-10-27 23:52:58,948 DEBUG [Listener at localhost/43373] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:58,949 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:58,950 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:58,950 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:58,950 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:58,951 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:52:58,951 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:58,951 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:52:58,952 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:52:59,200 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 2 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,201 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:52:59,201 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:52:59,201 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:52:59,203 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] 2023-10-27 23:52:59,204 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:52:59,204 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:52:59,204 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:52:59,204 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:52:59,205 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:52:59,208 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-10-27 23:52:59,209 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:52:59,210 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:52:59,210 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:52:59,210 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:52:59,211 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:59,211 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:59,212 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:59,212 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:59,212 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:52:59,213 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,213 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:52:59,213 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,213 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,213 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,214 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 8 msec 2023-10-27 23:52:59,214 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:52:59,215 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:52:59,216 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450809216 2023-10-27 23:52:59,217 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,217 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:52:59,221 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:52:59,222 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,36599,1698450767241, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241, openSeqNum=2 2023-10-27 23:52:59,222 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:52:59,222 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450839222 2023-10-27 23:52:59,222 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450899222 2023-10-27 23:52:59,222 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 4 msec 2023-10-27 23:52:59,222 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:52:59,224 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:52:59,225 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,225 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:52:59,225 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:52:59,472 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 3 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,472 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:52:59,473 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:52:59,473 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:52:59,473 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:52:59,473 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:52:59,473 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:52:59,473 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:52:59,473 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:52:59,474 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:52:59,476 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-10-27 23:52:59,477 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:52:59,477 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:52:59,478 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:52:59,478 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:52:59,478 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:59,479 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:59,479 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:59,479 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:59,480 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:52:59,480 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,480 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:52:59,480 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,481 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,481 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,481 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 6 msec 2023-10-27 23:52:59,482 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:52:59,482 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:52:59,482 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450809482 2023-10-27 23:52:59,484 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:52:59,484 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,487 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:52:59,487 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,36599,1698450767241, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241, openSeqNum=2 2023-10-27 23:52:59,487 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:52:59,488 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450839488 2023-10-27 23:52:59,488 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450899488 2023-10-27 23:52:59,488 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 4 msec 2023-10-27 23:52:59,488 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:52:59,491 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN}] 2023-10-27 23:52:59,491 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:52:59,491 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,491 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:52:59,491 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:52:59,738 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,738 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:52:59,738 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:52:59,739 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:52:59,739 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:52:59,739 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:52:59,739 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:52:59,739 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:52:59,739 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:52:59,740 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:52:59,742 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-10-27 23:52:59,743 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:52:59,743 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:52:59,743 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:52:59,744 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:52:59,745 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:59,745 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:52:59,745 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:59,746 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:52:59,746 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:52:59,747 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,747 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:52:59,747 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,747 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,747 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,747 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 6 msec 2023-10-27 23:52:59,748 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:52:59,749 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:52:59,749 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450809749 2023-10-27 23:52:59,751 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:52:59,751 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:52:59,754 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:52:59,754 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,36599,1698450767241, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241, openSeqNum=2 2023-10-27 23:52:59,754 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:52:59,754 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450839754 2023-10-27 23:52:59,755 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450899755 2023-10-27 23:52:59,755 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 4 msec 2023-10-27 23:52:59,755 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN}] 2023-10-27 23:52:59,755 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:52:59,758 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:52:59,759 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:52:59,759 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450779759"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450779759"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450779759"}]},"ts":"1698450779759"} 2023-10-27 23:52:59,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:52:59,761 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:52:59,761 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:52:59,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:52:59,761 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:00,005 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,005 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:00,005 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:00,005 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:00,006 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:00,006 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:00,006 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:00,006 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:00,006 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:00,007 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:00,008 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-10-27 23:53:00,008 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-10-27 23:53:00,009 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:00,009 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:00,009 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:00,010 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:00,010 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:00,010 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:00,010 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:00,010 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:00,011 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:00,012 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,012 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:53:00,012 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:00,012 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,012 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,012 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,012 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:00,012 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:53:00,012 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 5 msec 2023-10-27 23:53:00,013 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:00,013 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:00,014 INFO [Listener at localhost/43373] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab to restore RIT 2023-10-27 23:53:00,014 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450810014 2023-10-27 23:53:00,015 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:53:00,015 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:00,017 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:00,018 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=CLOSING, lastHost=jenkins-hbase16.apache.org,36599,1698450767241, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241, openSeqNum=2 2023-10-27 23:53:00,018 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:00,018 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450840018 2023-10-27 23:53:00,018 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450900018 2023-10-27 23:53:00,018 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:00,018 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:00,018 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450780018"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450780018"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450780018"}]},"ts":"1698450780018"} 2023-10-27 23:53:00,020 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:53:00,020 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:00,173 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:00,174 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,174 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing a6d7da66a8a7b852f46fc7477e45d6ab, disabling compactions & flushes 2023-10-27 23:53:00,174 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:53:00,174 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:53:00,174 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. after waiting 0 ms 2023-10-27 23:53:00,174 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:53:00,174 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing a6d7da66a8a7b852f46fc7477e45d6ab 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-10-27 23:53:00,185 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/394b355ebe7f4289bd9cac9ce1252ad1 2023-10-27 23:53:00,205 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/f3e823afc0e14d2da0f19eed70d76b23 2023-10-27 23:53:00,211 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf1/394b355ebe7f4289bd9cac9ce1252ad1 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1 2023-10-27 23:53:00,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:53:00,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/.tmp/cf2/f3e823afc0e14d2da0f19eed70d76b23 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23 2023-10-27 23:53:00,222 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:53:00,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for a6d7da66a8a7b852f46fc7477e45d6ab in 49ms, sequenceid=82, compaction requested=true 2023-10-27 23:53:00,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-10-27 23:53:00,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:53:00,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. 2023-10-27 23:53:00,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for a6d7da66a8a7b852f46fc7477e45d6ab: 2023-10-27 23:53:00,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,251 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=CLOSED 2023-10-27 23:53:00,251 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450780251"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450780251"}]},"ts":"1698450780251"} 2023-10-27 23:53:00,252 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:00,253 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=30, ppid=29, state=SUCCESS, locked=true; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:00,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:00,253 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=30, ppid=29, state=SUCCESS, locked=true; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:00,268 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,268 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:00,268 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:00,268 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:00,269 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:00,269 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:00,269 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:00,269 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:00,269 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:00,270 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:00,271 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-10-27 23:53:00,271 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-10-27 23:53:00,271 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4] 2023-10-27 23:53:00,271 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:00,272 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:00,272 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:00,272 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:00,272 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:00,273 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:00,273 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:00,273 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:00,273 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:00,274 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,274 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:53:00,274 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:00,274 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:00,275 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,275 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,275 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,275 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:00,275 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:53:00,275 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:00,275 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 5 msec 2023-10-27 23:53:00,276 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:00,276 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:00,276 INFO [Listener at localhost/43373] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab to restore RIT 2023-10-27 23:53:00,276 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450810276 2023-10-27 23:53:00,277 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:00,277 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:00,280 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:00,280 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,36599,1698450767241, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241, openSeqNum=2 2023-10-27 23:53:00,280 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:00,280 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450840280 2023-10-27 23:53:00,280 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450900280 2023-10-27 23:53:00,281 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:00,281 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=CLOSED, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:00,281 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450780281"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450780281"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450780281"}]},"ts":"1698450780281"} 2023-10-27 23:53:00,283 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:00,286 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=30, resume processing ppid=29 2023-10-27 23:53:00,287 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 in 263 msec 2023-10-27 23:53:00,287 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:00,287 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=29, ppid=28, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:53:00,287 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:00,287 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=29, ppid=28, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:00,438 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-27 23:53:00,531 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,531 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:00,531 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:00,531 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:00,532 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:00,532 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:00,532 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:00,532 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:00,532 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:00,533 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:00,534 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-10-27 23:53:00,534 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-10-27 23:53:00,534 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:00,535 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:00,535 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:00,536 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:00,536 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:00,536 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:00,536 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:00,536 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:00,537 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:00,537 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:00,537 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,537 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:53:00,538 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:00,538 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:00,538 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,538 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,538 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,538 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:00,538 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:00,538 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 5 msec 2023-10-27 23:53:00,539 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:00,539 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:00,540 INFO [Listener at localhost/43373] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab to restore RIT 2023-10-27 23:53:00,540 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450810540 2023-10-27 23:53:00,541 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:00,541 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:53:00,543 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:00,543 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,36599,1698450767241, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241, openSeqNum=2 2023-10-27 23:53:00,544 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:00,544 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450840544 2023-10-27 23:53:00,544 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450900544 2023-10-27 23:53:00,544 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:00,544 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:53:00,546 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:00,550 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=29, resume processing ppid=28 2023-10-27 23:53:00,550 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN in 791 msec 2023-10-27 23:53:00,552 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:00,552 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,552 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:00,552 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:00,794 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 4 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,794 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:00,794 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:00,794 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:00,795 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:00,795 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:00,795 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:00,795 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:00,795 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:00,796 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:00,797 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-10-27 23:53:00,797 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:53:00,797 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:00,797 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:00,798 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:00,798 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:00,798 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:00,798 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:00,798 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:00,799 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:00,799 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:00,799 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:00,800 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,800 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:53:00,800 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:00,800 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:00,800 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:00,800 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:00,800 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 4 msec 2023-10-27 23:53:00,801 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:00,802 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:00,802 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450810802 2023-10-27 23:53:00,803 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:00,803 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,806 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:00,806 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,36599,1698450767241, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241, openSeqNum=2 2023-10-27 23:53:00,806 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:00,807 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450840807 2023-10-27 23:53:00,807 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450900807 2023-10-27 23:53:00,807 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:00,807 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,807 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,807 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:00,813 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:00,822 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=28 splitting 14 storefiles, region=a6d7da66a8a7b852f46fc7477e45d6ab, threads=14 2023-10-27 23:53:00,823 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,823 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,823 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,823 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,823 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,824 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,824 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,824 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,824 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,824 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,824 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,825 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,825 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:00,825 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,007 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,035 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,035 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,035 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,039 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,059 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,059 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,067 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,067 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,071 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,072 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,075 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,075 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,077 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,077 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=28 split storefiles for region a6d7da66a8a7b852f46fc7477e45d6ab Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-10-27 23:53:01,114 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:01,114 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:01,114 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:01,115 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:01,307 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 5 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:01,307 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:01,307 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:01,307 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:01,308 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:01,308 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:01,308 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:01,308 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:01,308 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:01,309 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:01,310 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-10-27 23:53:01,310 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:53:01,310 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:01,310 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:01,311 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:01,311 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:01,311 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:01,312 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:01,312 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:01,312 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:01,312 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:01,313 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:01,313 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:01,314 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:53:01,314 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:01,314 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:01,314 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:01,314 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:01,314 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:01,314 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:01,314 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:01,314 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 5 msec 2023-10-27 23:53:01,315 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:01,315 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:01,316 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450811316 2023-10-27 23:53:01,317 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:01,317 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:01,319 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:01,320 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,36599,1698450767241, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241, openSeqNum=2 2023-10-27 23:53:01,320 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:01,320 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450841320 2023-10-27 23:53:01,320 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450901320 2023-10-27 23:53:01,320 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 2 msec 2023-10-27 23:53:01,329 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=28 splitting 14 storefiles, region=a6d7da66a8a7b852f46fc7477e45d6ab, threads=14 2023-10-27 23:53:01,330 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,330 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,330 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,330 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,331 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,331 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,331 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,331 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,331 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,331 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,335 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,335 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,342 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,342 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,508 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,551 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,551 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,562 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,570 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,579 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,580 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,591 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,596 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,603 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,603 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,604 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23 for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:01,604 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:02,012 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b for region: a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:02,012 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=28 split storefiles for region a6d7da66a8a7b852f46fc7477e45d6ab Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-10-27 23:53:02,058 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:02,063 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-10-27 23:53:02,065 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-10-27 23:53:02,065 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:02,065 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,065 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:02,065 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:02,070 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 6 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,070 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:02,070 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:02,071 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:02,071 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:02,071 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:02,071 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:02,071 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:02,071 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:02,072 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:02,073 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-10-27 23:53:02,074 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:53:02,074 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:02,074 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:02,074 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:02,075 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:02,075 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:02,075 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:02,076 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:02,076 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:02,076 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:02,077 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:02,077 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,077 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:53:02,077 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:02,078 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:02,078 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,078 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,078 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,078 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:02,078 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:02,078 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 6 msec 2023-10-27 23:53:02,079 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:02,080 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:02,080 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450812080 2023-10-27 23:53:02,081 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:02,081 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,084 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:02,084 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,36599,1698450767241, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241, openSeqNum=2 2023-10-27 23:53:02,085 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:02,085 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450842085 2023-10-27 23:53:02,085 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450902085 2023-10-27 23:53:02,085 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:02,088 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:02,090 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:02,090 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,090 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:02,090 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:02,335 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:53:02,335 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:02,336 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:02,336 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:02,336 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:02,336 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:02,336 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:02,336 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:02,336 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:02,337 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 1 msec 2023-10-27 23:53:02,339 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-10-27 23:53:02,339 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:53:02,339 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:02,339 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:02,340 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:02,340 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:02,340 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:02,341 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:02,341 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:02,341 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:02,342 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:02,342 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:02,343 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,343 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN 2023-10-27 23:53:02,343 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:02,343 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:02,343 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,343 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,344 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,344 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a6d7da66a8a7b852f46fc7477e45d6ab, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:02,344 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure a6d7da66a8a7b852f46fc7477e45d6ab, server=jenkins-hbase16.apache.org,36599,1698450767241 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:02,344 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 6 msec 2023-10-27 23:53:02,345 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:02,345 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:02,346 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450812346 2023-10-27 23:53:02,347 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,347 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:02,349 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:02,350 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=a6d7da66a8a7b852f46fc7477e45d6ab, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,36599,1698450767241, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241, openSeqNum=2 2023-10-27 23:53:02,350 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:02,350 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450842350 2023-10-27 23:53:02,350 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450902350 2023-10-27 23:53:02,350 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:02,354 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450782354"},{"qualifier":"splitA","vlen":56,"tag":[],"timestamp":"1698450782354"},{"qualifier":"splitB","vlen":56,"tag":[],"timestamp":"1698450782354"}]},"ts":"1698450782354"} 2023-10-27 23:53:02,354 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450782354"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450782354"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450782354"}]},"ts":"1698450782354"} 2023-10-27 23:53:02,354 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450782354"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450782354"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450782354"}]},"ts":"1698450782354"} 2023-10-27 23:53:02,360 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=409ad364226a99ff4e85bbe38f51653a, ASSIGN}, {pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=52ad738b07f186a78f22431d82fc4c1f, ASSIGN}] 2023-10-27 23:53:02,362 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=52ad738b07f186a78f22431d82fc4c1f, ASSIGN 2023-10-27 23:53:02,363 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=52ad738b07f186a78f22431d82fc4c1f, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase16.apache.org,36599,1698450767241; forceNewPlan=false, retain=false 2023-10-27 23:53:02,364 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=409ad364226a99ff4e85bbe38f51653a, ASSIGN 2023-10-27 23:53:02,365 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=409ad364226a99ff4e85bbe38f51653a, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase16.apache.org,36599,1698450767241; forceNewPlan=false, retain=false 2023-10-27 23:53:02,513 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-10-27 23:53:02,514 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=409ad364226a99ff4e85bbe38f51653a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:02,515 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450782514"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450782514"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450782514"}]},"ts":"1698450782514"} 2023-10-27 23:53:02,517 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=33, ppid=31, state=RUNNABLE; OpenRegionProcedure 409ad364226a99ff4e85bbe38f51653a, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:53:02,520 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=52ad738b07f186a78f22431d82fc4c1f, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:02,521 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450782520"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450782520"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450782520"}]},"ts":"1698450782520"} 2023-10-27 23:53:02,522 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=34, ppid=32, state=RUNNABLE; OpenRegionProcedure 52ad738b07f186a78f22431d82fc4c1f, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:53:02,669 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:02,673 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a. 2023-10-27 23:53:02,673 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 409ad364226a99ff4e85bbe38f51653a, NAME => 'testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.', STARTKEY => '', ENDKEY => '41'} 2023-10-27 23:53:02,674 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:53:02,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR 409ad364226a99ff4e85bbe38f51653a 2023-10-27 23:53:02,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:02,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 409ad364226a99ff4e85bbe38f51653a 2023-10-27 23:53:02,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 409ad364226a99ff4e85bbe38f51653a 2023-10-27 23:53:02,675 INFO [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 409ad364226a99ff4e85bbe38f51653a 2023-10-27 23:53:02,676 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1 2023-10-27 23:53:02,676 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1 2023-10-27 23:53:02,677 INFO [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 409ad364226a99ff4e85bbe38f51653a columnFamilyName cf1 2023-10-27 23:53:02,697 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1-bottom 2023-10-27 23:53:02,703 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9-bottom 2023-10-27 23:53:02,709 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e-bottom 2023-10-27 23:53:02,715 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795-bottom 2023-10-27 23:53:02,721 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f-bottom 2023-10-27 23:53:02,727 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190-bottom 2023-10-27 23:53:02,734 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b-bottom 2023-10-27 23:53:02,734 INFO [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(310): Store=409ad364226a99ff4e85bbe38f51653a/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:02,734 INFO [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 409ad364226a99ff4e85bbe38f51653a 2023-10-27 23:53:02,735 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2 2023-10-27 23:53:02,735 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2 2023-10-27 23:53:02,736 INFO [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 409ad364226a99ff4e85bbe38f51653a columnFamilyName cf2 2023-10-27 23:53:02,753 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7-bottom 2023-10-27 23:53:02,759 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b-bottom 2023-10-27 23:53:02,765 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902-bottom 2023-10-27 23:53:02,770 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b-bottom 2023-10-27 23:53:02,775 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1-bottom 2023-10-27 23:53:02,780 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5-bottom 2023-10-27 23:53:02,785 DEBUG [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23-bottom 2023-10-27 23:53:02,786 INFO [StoreOpener-409ad364226a99ff4e85bbe38f51653a-1] regionserver.HStore(310): Store=409ad364226a99ff4e85bbe38f51653a/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:02,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a 2023-10-27 23:53:02,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a 2023-10-27 23:53:02,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:53:02,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 409ad364226a99ff4e85bbe38f51653a 2023-10-27 23:53:02,791 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 409ad364226a99ff4e85bbe38f51653a; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67301197, jitterRate=0.0028659850358963013}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:53:02,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 409ad364226a99ff4e85bbe38f51653a: 2023-10-27 23:53:02,792 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a., pid=33, masterSystemTime=1698450782669 2023-10-27 23:53:02,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:53:02,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:53:02,793 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:53:02,793 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:53:02,793 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a. 2023-10-27 23:53:02,793 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a. 2023-10-27 23:53:02,793 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1912): 409ad364226a99ff4e85bbe38f51653a/cf1 is initiating minor compaction (all files) 2023-10-27 23:53:02,793 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1912): 409ad364226a99ff4e85bbe38f51653a/cf2 is initiating minor compaction (all files) 2023-10-27 23:53:02,793 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 409ad364226a99ff4e85bbe38f51653a/cf1 in testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a. 2023-10-27 23:53:02,793 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 409ad364226a99ff4e85bbe38f51653a/cf2 in testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a. 2023-10-27 23:53:02,794 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1-bottom] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/.tmp, totalSize=34.9 K 2023-10-27 23:53:02,794 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23-bottom] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/.tmp, totalSize=34.9 K 2023-10-27 23:53:02,795 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450777107 2023-10-27 23:53:02,795 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450777107 2023-10-27 23:53:02,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a. 2023-10-27 23:53:02,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a. 2023-10-27 23:53:02,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f. 2023-10-27 23:53:02,795 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450777361 2023-10-27 23:53:02,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 52ad738b07f186a78f22431d82fc4c1f, NAME => 'testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.', STARTKEY => '41', ENDKEY => ''} 2023-10-27 23:53:02,795 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450777361 2023-10-27 23:53:02,796 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:53:02,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR 52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,796 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450777623 2023-10-27 23:53:02,796 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=409ad364226a99ff4e85bbe38f51653a, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:02,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:02,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,796 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450782796"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450782796"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450782796"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450782796"}]},"ts":"1698450782796"} 2023-10-27 23:53:02,796 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450777623 2023-10-27 23:53:02,797 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting 1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450777874 2023-10-27 23:53:02,798 INFO [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,798 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting 5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450778140 2023-10-27 23:53:02,798 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450777874 2023-10-27 23:53:02,799 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting 51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450778412 2023-10-27 23:53:02,799 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450778140 2023-10-27 23:53:02,799 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1 2023-10-27 23:53:02,799 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1 2023-10-27 23:53:02,799 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1698450778681 2023-10-27 23:53:02,800 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450778412 2023-10-27 23:53:02,800 INFO [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 52ad738b07f186a78f22431d82fc4c1f columnFamilyName cf1 2023-10-27 23:53:02,800 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1698450778681 2023-10-27 23:53:02,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=33, resume processing ppid=31 2023-10-27 23:53:02,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=33, ppid=31, state=SUCCESS; OpenRegionProcedure 409ad364226a99ff4e85bbe38f51653a, server=jenkins-hbase16.apache.org,36599,1698450767241 in 282 msec 2023-10-27 23:53:02,804 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=31, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=409ad364226a99ff4e85bbe38f51653a, ASSIGN in 443 msec 2023-10-27 23:53:02,817 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 409ad364226a99ff4e85bbe38f51653a#cf1#compaction#32 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:02,819 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] throttle.PressureAwareThroughputController(145): 409ad364226a99ff4e85bbe38f51653a#cf2#compaction#33 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:02,830 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1-top 2023-10-27 23:53:02,835 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9-top 2023-10-27 23:53:02,845 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e-top 2023-10-27 23:53:02,852 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795-top 2023-10-27 23:53:02,856 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/.tmp/cf1/c8be6bc780304a31a72b864badb0cb86 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/c8be6bc780304a31a72b864badb0cb86 2023-10-27 23:53:02,857 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f-top 2023-10-27 23:53:02,858 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/.tmp/cf2/e8f238648fc8425da2ba64857c2dca03 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/e8f238648fc8425da2ba64857c2dca03 2023-10-27 23:53:02,863 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 409ad364226a99ff4e85bbe38f51653a/cf1 of 409ad364226a99ff4e85bbe38f51653a into c8be6bc780304a31a72b864badb0cb86(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:02,863 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 409ad364226a99ff4e85bbe38f51653a: 2023-10-27 23:53:02,864 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a., storeName=409ad364226a99ff4e85bbe38f51653a/cf1, priority=9, startTime=1698450782792; duration=0sec 2023-10-27 23:53:02,864 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:02,869 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190-top 2023-10-27 23:53:02,870 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 409ad364226a99ff4e85bbe38f51653a/cf2 of 409ad364226a99ff4e85bbe38f51653a into e8f238648fc8425da2ba64857c2dca03(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:02,870 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 409ad364226a99ff4e85bbe38f51653a: 2023-10-27 23:53:02,870 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a., storeName=409ad364226a99ff4e85bbe38f51653a/cf2, priority=9, startTime=1698450782793; duration=0sec 2023-10-27 23:53:02,871 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:02,874 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b-top 2023-10-27 23:53:02,874 INFO [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(310): Store=52ad738b07f186a78f22431d82fc4c1f/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:02,874 INFO [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,876 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2 2023-10-27 23:53:02,876 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2 2023-10-27 23:53:02,876 INFO [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 52ad738b07f186a78f22431d82fc4c1f columnFamilyName cf2 2023-10-27 23:53:02,895 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7-top 2023-10-27 23:53:02,900 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b-top 2023-10-27 23:53:02,905 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902-top 2023-10-27 23:53:02,911 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b-top 2023-10-27 23:53:02,917 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1-top 2023-10-27 23:53:02,922 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5-top 2023-10-27 23:53:02,930 DEBUG [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23-top 2023-10-27 23:53:02,931 INFO [StoreOpener-52ad738b07f186a78f22431d82fc4c1f-1] regionserver.HStore(310): Store=52ad738b07f186a78f22431d82fc4c1f/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:02,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:53:02,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:02,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 52ad738b07f186a78f22431d82fc4c1f; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73607372, jitterRate=0.09683531522750854}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:53:02,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 52ad738b07f186a78f22431d82fc4c1f: 2023-10-27 23:53:02,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f., pid=34, masterSystemTime=1698450782669 2023-10-27 23:53:02,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:53:02,940 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:53:02,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:53:02,940 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:53:02,940 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f. 2023-10-27 23:53:02,940 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f. 2023-10-27 23:53:02,940 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1912): 52ad738b07f186a78f22431d82fc4c1f/cf1 is initiating minor compaction (all files) 2023-10-27 23:53:02,940 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1912): 52ad738b07f186a78f22431d82fc4c1f/cf2 is initiating minor compaction (all files) 2023-10-27 23:53:02,940 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52ad738b07f186a78f22431d82fc4c1f/cf1 in testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f. 2023-10-27 23:53:02,940 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52ad738b07f186a78f22431d82fc4c1f/cf2 in testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f. 2023-10-27 23:53:02,940 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23-top] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/.tmp, totalSize=34.9 K 2023-10-27 23:53:02,941 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1-top] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/.tmp, totalSize=34.9 K 2023-10-27 23:53:02,941 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1698450777107 2023-10-27 23:53:02,941 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1698450777107 2023-10-27 23:53:02,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f. 2023-10-27 23:53:02,942 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f. 2023-10-27 23:53:02,942 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1698450777361 2023-10-27 23:53:02,942 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1698450777361 2023-10-27 23:53:02,942 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=52ad738b07f186a78f22431d82fc4c1f, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:02,942 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1698450777623 2023-10-27 23:53:02,942 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450782942"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450782942"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450782942"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450782942"}]},"ts":"1698450782942"} 2023-10-27 23:53:02,943 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1698450777623 2023-10-27 23:53:02,943 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting 1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1698450777874 2023-10-27 23:53:02,943 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1698450777874 2023-10-27 23:53:02,943 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting 5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1698450778140 2023-10-27 23:53:02,943 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1698450778140 2023-10-27 23:53:02,944 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting 51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1698450778412 2023-10-27 23:53:02,944 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1698450778412 2023-10-27 23:53:02,944 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] compactions.Compactor(207): Compacting f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1698450778681 2023-10-27 23:53:02,944 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] compactions.Compactor(207): Compacting 394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1698450778681 2023-10-27 23:53:02,947 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=34, resume processing ppid=32 2023-10-27 23:53:02,947 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=34, ppid=32, state=SUCCESS; OpenRegionProcedure 52ad738b07f186a78f22431d82fc4c1f, server=jenkins-hbase16.apache.org,36599,1698450767241 in 422 msec 2023-10-27 23:53:02,950 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=32, resume processing ppid=28 2023-10-27 23:53:02,950 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=32, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=52ad738b07f186a78f22431d82fc4c1f, ASSIGN in 588 msec 2023-10-27 23:53:02,953 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f in 4.0020 sec 2023-10-27 23:53:02,961 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52ad738b07f186a78f22431d82fc4c1f#cf2#compaction#34 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:02,961 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52ad738b07f186a78f22431d82fc4c1f#cf1#compaction#35 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:02,990 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/.tmp/cf2/d2c7be4edab44eac9e3f3001b6f79738 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/d2c7be4edab44eac9e3f3001b6f79738 2023-10-27 23:53:02,998 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/.tmp/cf1/26e0f5ff2a6f4a47a0f6b2554ca3ca35 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/26e0f5ff2a6f4a47a0f6b2554ca3ca35 2023-10-27 23:53:02,998 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 52ad738b07f186a78f22431d82fc4c1f/cf2 of 52ad738b07f186a78f22431d82fc4c1f into d2c7be4edab44eac9e3f3001b6f79738(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:02,998 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52ad738b07f186a78f22431d82fc4c1f: 2023-10-27 23:53:02,998 INFO [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f., storeName=52ad738b07f186a78f22431d82fc4c1f/cf2, priority=9, startTime=1698450782940; duration=0sec 2023-10-27 23:53:02,998 DEBUG [RS:2;jenkins-hbase16:36599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:03,004 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 52ad738b07f186a78f22431d82fc4c1f/cf1 of 52ad738b07f186a78f22431d82fc4c1f into 26e0f5ff2a6f4a47a0f6b2554ca3ca35(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:03,004 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52ad738b07f186a78f22431d82fc4c1f: 2023-10-27 23:53:03,004 INFO [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f., storeName=52ad738b07f186a78f22431d82fc4c1f/cf1, priority=9, startTime=1698450782939; duration=0sec 2023-10-27 23:53:03,004 DEBUG [RS:2;jenkins-hbase16:36599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:03,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36599] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Scan size: 143 connection: 188.40.62.62:49794 deadline: 1698450843102, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab. is not online on jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:03,216 DEBUG [Listener at localhost/43373] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-10-27 23:53:03,225 INFO [Listener at localhost/43373] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-10-27 23:53:03,228 DEBUG [Listener at localhost/43373] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-10-27 23:53:03,248 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:53:03,250 INFO [Listener at localhost/43373] client.HBaseAdmin$15(890): Started disable of testSplitWithoutPONR 2023-10-27 23:53:03,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable testSplitWithoutPONR 2023-10-27 23:53:03,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=35, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:03,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=35 2023-10-27 23:53:03,255 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450783255"}]},"ts":"1698450783255"} 2023-10-27 23:53:03,256 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=DISABLING in hbase:meta 2023-10-27 23:53:03,322 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitWithoutPONR to state=DISABLING 2023-10-27 23:53:03,323 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=409ad364226a99ff4e85bbe38f51653a, UNASSIGN}, {pid=37, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=52ad738b07f186a78f22431d82fc4c1f, UNASSIGN}] 2023-10-27 23:53:03,324 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=52ad738b07f186a78f22431d82fc4c1f, UNASSIGN 2023-10-27 23:53:03,325 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=52ad738b07f186a78f22431d82fc4c1f, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:03,325 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450783325"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450783325"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450783325"}]},"ts":"1698450783325"} 2023-10-27 23:53:03,327 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=38, ppid=37, state=RUNNABLE; CloseRegionProcedure 52ad738b07f186a78f22431d82fc4c1f, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:53:03,329 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=409ad364226a99ff4e85bbe38f51653a, UNASSIGN 2023-10-27 23:53:03,330 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=409ad364226a99ff4e85bbe38f51653a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:03,330 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450783330"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450783330"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450783330"}]},"ts":"1698450783330"} 2023-10-27 23:53:03,332 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=39, ppid=36, state=RUNNABLE; CloseRegionProcedure 409ad364226a99ff4e85bbe38f51653a, server=jenkins-hbase16.apache.org,36599,1698450767241}] 2023-10-27 23:53:03,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=35 2023-10-27 23:53:03,478 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:03,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:03,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 52ad738b07f186a78f22431d82fc4c1f, disabling compactions & flushes 2023-10-27 23:53:03,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f. 2023-10-27 23:53:03,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f. 2023-10-27 23:53:03,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f. after waiting 0 ms 2023-10-27 23:53:03,480 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f. 2023-10-27 23:53:03,490 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1-top] to archive 2023-10-27 23:53:03,491 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:53:03,492 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,494 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,495 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,497 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,498 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,500 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,501 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,503 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23-top] to archive 2023-10-27 23:53:03,504 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:53:03,506 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,507 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,508 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,510 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,511 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,512 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,513 DEBUG [StoreCloser-testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-10-27 23:53:03,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:53:03,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f. 2023-10-27 23:53:03,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 52ad738b07f186a78f22431d82fc4c1f: 2023-10-27 23:53:03,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:03,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 409ad364226a99ff4e85bbe38f51653a 2023-10-27 23:53:03,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 409ad364226a99ff4e85bbe38f51653a, disabling compactions & flushes 2023-10-27 23:53:03,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a. 2023-10-27 23:53:03,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a. 2023-10-27 23:53:03,521 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=52ad738b07f186a78f22431d82fc4c1f, regionState=CLOSED 2023-10-27 23:53:03,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a. after waiting 0 ms 2023-10-27 23:53:03,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a. 2023-10-27 23:53:03,521 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450783521"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450783521"}]},"ts":"1698450783521"} 2023-10-27 23:53:03,522 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1-bottom] to archive 2023-10-27 23:53:03,523 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:53:03,525 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=38, resume processing ppid=37 2023-10-27 23:53:03,526 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=38, ppid=37, state=SUCCESS; CloseRegionProcedure 52ad738b07f186a78f22431d82fc4c1f, server=jenkins-hbase16.apache.org,36599,1698450767241 in 196 msec 2023-10-27 23:53:03,527 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,528 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=37, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=52ad738b07f186a78f22431d82fc4c1f, UNASSIGN in 203 msec 2023-10-27 23:53:03,529 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,530 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,532 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,533 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,534 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,536 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,538 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23-bottom] to archive 2023-10-27 23:53:03,539 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:53:03,540 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,541 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,542 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,543 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,545 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,546 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,547 DEBUG [StoreCloser-testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-10-27 23:53:03,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:53:03,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a. 2023-10-27 23:53:03,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 409ad364226a99ff4e85bbe38f51653a: 2023-10-27 23:53:03,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 409ad364226a99ff4e85bbe38f51653a 2023-10-27 23:53:03,554 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=409ad364226a99ff4e85bbe38f51653a, regionState=CLOSED 2023-10-27 23:53:03,554 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450783554"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450783554"}]},"ts":"1698450783554"} 2023-10-27 23:53:03,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=35 2023-10-27 23:53:03,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=39, resume processing ppid=36 2023-10-27 23:53:03,558 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=39, ppid=36, state=SUCCESS; CloseRegionProcedure 409ad364226a99ff4e85bbe38f51653a, server=jenkins-hbase16.apache.org,36599,1698450767241 in 224 msec 2023-10-27 23:53:03,560 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=36, resume processing ppid=35 2023-10-27 23:53:03,560 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=409ad364226a99ff4e85bbe38f51653a, UNASSIGN in 235 msec 2023-10-27 23:53:03,561 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450783561"}]},"ts":"1698450783561"} 2023-10-27 23:53:03,562 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=DISABLED in hbase:meta 2023-10-27 23:53:03,574 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitWithoutPONR to state=DISABLED 2023-10-27 23:53:03,577 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR in 325 msec 2023-10-27 23:53:03,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=35 2023-10-27 23:53:03,859 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitWithoutPONR, procId: 35 completed 2023-10-27 23:53:03,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete testSplitWithoutPONR 2023-10-27 23:53:03,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=40, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:03,863 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=40, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:03,864 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=40, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:03,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=40 2023-10-27 23:53:03,871 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,871 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a 2023-10-27 23:53:03,871 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:03,873 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/recovered.edits] 2023-10-27 23:53:03,873 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/recovered.edits] 2023-10-27 23:53:03,874 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/recovered.edits] 2023-10-27 23:53:03,877 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/.regioninfo to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/.regioninfo 2023-10-27 23:53:03,877 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/.regioninfo to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/.regioninfo 2023-10-27 23:53:03,877 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/394b355ebe7f4289bd9cac9ce1252ad1 2023-10-27 23:53:03,879 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4609a3ebc1eb4c9283d184a3e62f39e9 2023-10-27 23:53:03,879 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,880 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/394b355ebe7f4289bd9cac9ce1252ad1.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,880 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e 2023-10-27 23:53:03,881 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,881 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/4609a3ebc1eb4c9283d184a3e62f39e9.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,881 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/62d169ea60984148b3a4bf88cfa16795 2023-10-27 23:53:03,882 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,882 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/4b9e9bb0e3a443d7b0293b3fd5f2c48e.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,883 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/6bb4b637cac14a1788c8db687836756f 2023-10-27 23:53:03,884 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,884 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/62d169ea60984148b3a4bf88cfa16795.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,884 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/b0c83c29a88d4cfb872b9cd9720c6190 2023-10-27 23:53:03,885 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,885 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/6bb4b637cac14a1788c8db687836756f.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,886 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf1/f2dfa3131f604f7588765386c7a0da7b 2023-10-27 23:53:03,886 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,886 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/b0c83c29a88d4cfb872b9cd9720c6190.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,887 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,888 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf1/f2dfa3131f604f7588765386c7a0da7b.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,888 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/1c24b0739cc246ff8abbb83025837bf7 2023-10-27 23:53:03,890 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/51df109353864f6daa7599a3e1a1ed2b 2023-10-27 23:53:03,891 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,891 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/1c24b0739cc246ff8abbb83025837bf7.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,892 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/5aa45f7e575a46fc8befa178336ab902 2023-10-27 23:53:03,892 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,893 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/51df109353864f6daa7599a3e1a1ed2b.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,893 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/aefddd22bc754dc5b3eca560340c2f8b 2023-10-27 23:53:03,894 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,894 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/5aa45f7e575a46fc8befa178336ab902.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,895 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/c9a2e59aa69e4f758834d00a2f4796a1 2023-10-27 23:53:03,896 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,896 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/aefddd22bc754dc5b3eca560340c2f8b.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,897 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/de202d94265f41549c744b6d939d1fb5 2023-10-27 23:53:03,897 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,898 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/c9a2e59aa69e4f758834d00a2f4796a1.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,899 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,899 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/cf2/f3e823afc0e14d2da0f19eed70d76b23 2023-10-27 23:53:03,900 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/de202d94265f41549c744b6d939d1fb5.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,900 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/52ad738b07f186a78f22431d82fc4c1f/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,902 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/409ad364226a99ff4e85bbe38f51653a/cf2/f3e823afc0e14d2da0f19eed70d76b23.a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,903 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/recovered.edits/85.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab/recovered.edits/85.seqid 2023-10-27 23:53:03,904 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/26e0f5ff2a6f4a47a0f6b2554ca3ca35 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf1/26e0f5ff2a6f4a47a0f6b2554ca3ca35 2023-10-27 23:53:03,904 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/a6d7da66a8a7b852f46fc7477e45d6ab 2023-10-27 23:53:03,907 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/d2c7be4edab44eac9e3f3001b6f79738 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/cf2/d2c7be4edab44eac9e3f3001b6f79738 2023-10-27 23:53:03,911 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/recovered.edits/90.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f/recovered.edits/90.seqid 2023-10-27 23:53:03,911 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/c8be6bc780304a31a72b864badb0cb86 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf1/c8be6bc780304a31a72b864badb0cb86 2023-10-27 23:53:03,912 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:03,915 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/e8f238648fc8425da2ba64857c2dca03 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/cf2/e8f238648fc8425da2ba64857c2dca03 2023-10-27 23:53:03,918 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/recovered.edits/90.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a/recovered.edits/90.seqid 2023-10-27 23:53:03,919 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitWithoutPONR/409ad364226a99ff4e85bbe38f51653a 2023-10-27 23:53:03,919 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitWithoutPONR regions 2023-10-27 23:53:03,922 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=40, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:03,925 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitWithoutPONR from hbase:meta 2023-10-27 23:53:03,928 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitWithoutPONR' descriptor. 2023-10-27 23:53:03,930 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=40, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:03,930 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitWithoutPONR' from region states. 2023-10-27 23:53:03,930 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450783930"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:03,930 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450783930"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:03,930 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450783930"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:03,933 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-10-27 23:53:03,933 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a6d7da66a8a7b852f46fc7477e45d6ab, NAME => 'testSplitWithoutPONR,,1698450776602.a6d7da66a8a7b852f46fc7477e45d6ab.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 409ad364226a99ff4e85bbe38f51653a, NAME => 'testSplitWithoutPONR,,1698450778948.409ad364226a99ff4e85bbe38f51653a.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => 52ad738b07f186a78f22431d82fc4c1f, NAME => 'testSplitWithoutPONR,41,1698450778948.52ad738b07f186a78f22431d82fc4c1f.', STARTKEY => '41', ENDKEY => ''}] 2023-10-27 23:53:03,933 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitWithoutPONR' as deleted. 2023-10-27 23:53:03,934 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1698450783933"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:03,936 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitWithoutPONR state from META 2023-10-27 23:53:03,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=40 2023-10-27 23:53:04,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=40 2023-10-27 23:53:04,179 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=40, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:04,180 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR in 318 msec 2023-10-27 23:53:04,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=40 2023-10-27 23:53:04,469 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitWithoutPONR, procId: 40 completed 2023-10-27 23:53:04,482 INFO [Listener at localhost/43373] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitWithoutPONR Thread=397 (was 392) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:55048 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:37342 [Waiting for operation #14] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x27f582be-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:43714 [Waiting for operation #19] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:37356 [Waiting for operation #13] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:43512 [Waiting for operation #16] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:43558 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:57432 [Waiting for operation #23] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:37166 [Waiting for operation #23] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:37296 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-metaLookup-shared--pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:43548 [Waiting for operation #18] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:55076 [Waiting for operation #27] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:37364 [Waiting for operation #21] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:43698 [Waiting for operation #13] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x27f582be-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:37924 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:55046 [Waiting for operation #18] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-24173407_17 at /127.0.0.1:44476 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=844 (was 872), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=376 (was 400), ProcessCount=170 (was 170), AvailableMemoryMB=5670 (was 5732) 2023-10-27 23:53:04,498 INFO [Listener at localhost/43373] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRollbackAndDoubleExecution Thread=397, OpenFileDescriptor=844, MaxFileDescriptor=60000, SystemLoadAverage=376, ProcessCount=170, AvailableMemoryMB=5667 2023-10-27 23:53:04,499 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:53:04,533 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-10-27 23:53:04,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(492): Client=jenkins//188.40.62.62 set balanceSwitch=false 2023-10-27 23:53:04,535 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=41, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:04,537 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_PRE_OPERATION 2023-10-27 23:53:04,537 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-10-27 23:53:04,539 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:04,540 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5 empty. 2023-10-27 23:53:04,540 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:04,540 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackAndDoubleExecution regions 2023-10-27 23:53:04,551 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackAndDoubleExecution/.tabledesc/.tableinfo.0000000001 2023-10-27 23:53:04,553 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(7675): creating {ENCODED => 38d29e3b4a5d0b766af1d906cf63bcf5, NAME => 'testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRollbackAndDoubleExecution', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp 2023-10-27 23:53:04,561 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(866): Instantiated testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:04,561 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1604): Closing 38d29e3b4a5d0b766af1d906cf63bcf5, disabling compactions & flushes 2023-10-27 23:53:04,561 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1626): Closing region testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:04,562 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:04,562 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1714): Acquired close lock on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. after waiting 0 ms 2023-10-27 23:53:04,562 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1724): Updates disabled for region testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:04,562 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1838): Closed testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:04,562 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1558): Region close journal for 38d29e3b4a5d0b766af1d906cf63bcf5: 2023-10-27 23:53:04,564 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_ADD_TO_META 2023-10-27 23:53:04,565 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450784565"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450784565"}]},"ts":"1698450784565"} 2023-10-27 23:53:04,567 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-10-27 23:53:04,568 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-10-27 23:53:04,568 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450784568"}]},"ts":"1698450784568"} 2023-10-27 23:53:04,569 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=ENABLING in hbase:meta 2023-10-27 23:53:04,617 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-10-27 23:53:04,619 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-27 23:53:04,619 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-10-27 23:53:04,619 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-10-27 23:53:04,619 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-10-27 23:53:04,619 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN}] 2023-10-27 23:53:04,621 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:04,622 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,43641,1698450767195; forceNewPlan=false, retain=false 2023-10-27 23:53:04,772 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-10-27 23:53:04,773 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:04,773 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450784773"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450784773"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450784773"}]},"ts":"1698450784773"} 2023-10-27 23:53:04,775 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE; OpenRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:53:04,928 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:04,928 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-10-27 23:53:04,932 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:48686, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-10-27 23:53:04,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:04,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 38d29e3b4a5d0b766af1d906cf63bcf5, NAME => 'testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:53:04,940 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:53:04,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:04,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:04,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:04,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:04,943 INFO [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:04,945 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1 2023-10-27 23:53:04,945 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1 2023-10-27 23:53:04,945 INFO [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38d29e3b4a5d0b766af1d906cf63bcf5 columnFamilyName cf1 2023-10-27 23:53:04,946 INFO [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(310): Store=38d29e3b4a5d0b766af1d906cf63bcf5/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:04,946 INFO [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:04,947 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2 2023-10-27 23:53:04,948 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2 2023-10-27 23:53:04,948 INFO [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38d29e3b4a5d0b766af1d906cf63bcf5 columnFamilyName cf2 2023-10-27 23:53:04,949 INFO [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(310): Store=38d29e3b4a5d0b766af1d906cf63bcf5/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:04,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:04,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:04,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRollbackAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:53:04,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:04,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-10-27 23:53:04,957 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 38d29e3b4a5d0b766af1d906cf63bcf5; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63781930, jitterRate=-0.04957517981529236}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:53:04,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 38d29e3b4a5d0b766af1d906cf63bcf5: 2023-10-27 23:53:04,959 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5., pid=43, masterSystemTime=1698450784928 2023-10-27 23:53:04,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:04,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:04,963 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:04,963 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450784963"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450784963"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450784963"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450784963"}]},"ts":"1698450784963"} 2023-10-27 23:53:04,967 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=43, resume processing ppid=42 2023-10-27 23:53:04,967 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=43, ppid=42, state=SUCCESS; OpenRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 in 190 msec 2023-10-27 23:53:04,969 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=42, resume processing ppid=41 2023-10-27 23:53:04,969 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=42, ppid=41, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN in 348 msec 2023-10-27 23:53:04,970 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-10-27 23:53:04,970 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450784970"}]},"ts":"1698450784970"} 2023-10-27 23:53:04,972 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=ENABLED in hbase:meta 2023-10-27 23:53:04,984 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_POST_OPERATION 2023-10-27 23:53:04,986 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution in 450 msec 2023-10-27 23:53:05,038 DEBUG [Listener at localhost/43373] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-10-27 23:53:05,043 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:48696, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-10-27 23:53:05,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:05,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-10-27 23:53:05,052 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-10-27 23:53:05,053 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,053 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-10-27 23:53:05,053 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-10-27 23:53:05,053 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,053 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,061 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,061 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,061 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,061 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,061 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,062 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:05,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:05,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:05,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:05,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:05,063 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:05,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:05,063 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,063 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:05,063 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:05,063 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,063 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:05,063 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:05,063 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:05,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:05,063 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:05,063 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,063 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:05,063 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:05,063 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:05,064 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:05,063 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. started... 2023-10-27 23:53:05,063 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:05,064 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:05,064 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38d29e3b4a5d0b766af1d906cf63bcf5 2/2 column families, dataSize=120 B heapSize=928 B 2023-10-27 23:53:05,074 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,076 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/2defc9652d4341c6aad9dbeaac483c78 2023-10-27 23:53:05,082 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,082 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,082 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:05,083 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,083 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:05,090 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/21df8aa0c3dd4184906a6001d2632643 2023-10-27 23:53:05,095 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/2defc9652d4341c6aad9dbeaac483c78 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/2defc9652d4341c6aad9dbeaac483c78 2023-10-27 23:53:05,100 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/2defc9652d4341c6aad9dbeaac483c78, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:53:05,100 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/21df8aa0c3dd4184906a6001d2632643 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/21df8aa0c3dd4184906a6001d2632643 2023-10-27 23:53:05,105 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/21df8aa0c3dd4184906a6001d2632643, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:53:05,106 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 38d29e3b4a5d0b766af1d906cf63bcf5 in 42ms, sequenceid=6, compaction requested=false 2023-10-27 23:53:05,106 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRollbackAndDoubleExecution' 2023-10-27 23:53:05,107 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38d29e3b4a5d0b766af1d906cf63bcf5: 2023-10-27 23:53:05,107 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:05,107 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:05,107 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:05,107 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,107 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:05,107 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:05,116 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,116 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:05,116 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,116 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:05,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-10-27 23:53:05,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65d5907[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:05,118 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-10-27 23:53:05,118 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,127 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,127 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,127 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,128 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,128 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,128 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:05,128 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:05,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,128 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:05,128 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:05,128 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:05,128 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:05,128 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:05,129 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:05,129 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:05,136 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:05,136 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:05,136 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,136 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:05,136 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,136 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:05,144 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:05,144 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,144 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,144 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:05,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:53:05,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-10-27 23:53:05,148 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-10-27 23:53:05,148 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:05,148 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:05,148 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-10-27 23:53:05,148 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:05,158 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,158 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,158 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,158 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,158 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,158 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,158 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:05,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,158 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,159 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,159 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,161 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,161 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:05,161 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,169 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,169 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,169 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,169 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,170 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-10-27 23:53:05,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:05,170 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,170 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,169 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,170 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,170 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:05,170 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,170 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:05,170 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,171 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,171 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,171 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,171 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,171 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,171 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,171 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,271 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:05,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:05,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:05,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-10-27 23:53:05,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-10-27 23:53:05,291 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-10-27 23:53:05,291 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,291 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-10-27 23:53:05,291 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-10-27 23:53:05,292 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,292 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,303 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,303 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,303 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,303 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,303 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,303 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:05,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:05,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:05,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:05,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:05,305 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:05,305 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. started... 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:05,305 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:05,305 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38d29e3b4a5d0b766af1d906cf63bcf5 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:53:05,316 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,316 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,316 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,316 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:05,316 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,316 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:05,321 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/76f1e2d8901440ce89125dbf208fcfa0 2023-10-27 23:53:05,337 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/22ccc037a33e47f0a14a23f59f65ec95 2023-10-27 23:53:05,341 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/76f1e2d8901440ce89125dbf208fcfa0 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/76f1e2d8901440ce89125dbf208fcfa0 2023-10-27 23:53:05,347 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/76f1e2d8901440ce89125dbf208fcfa0, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:53:05,348 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/22ccc037a33e47f0a14a23f59f65ec95 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/22ccc037a33e47f0a14a23f59f65ec95 2023-10-27 23:53:05,352 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/22ccc037a33e47f0a14a23f59f65ec95, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:53:05,353 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 38d29e3b4a5d0b766af1d906cf63bcf5 in 48ms, sequenceid=19, compaction requested=false 2023-10-27 23:53:05,353 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38d29e3b4a5d0b766af1d906cf63bcf5: 2023-10-27 23:53:05,353 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:05,353 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:05,353 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:05,353 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,353 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:05,353 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:05,361 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,361 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:05,361 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,361 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:05,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-10-27 23:53:05,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d13bb2e[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:05,364 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-10-27 23:53:05,364 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,374 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,374 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,374 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,375 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:05,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,374 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,375 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:05,375 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:05,375 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:05,375 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:05,375 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:05,375 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:05,375 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:05,375 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:05,375 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:05,405 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:05,405 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,405 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:05,405 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,406 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,406 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:05,406 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:05,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:05,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:05,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:05,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:05,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:05,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:53:05,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-10-27 23:53:05,419 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-10-27 23:53:05,419 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:05,419 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:05,419 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-10-27 23:53:05,419 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:05,428 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,428 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,428 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,428 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,428 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,428 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,428 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:05,428 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:05,429 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,429 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:05,431 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,431 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,431 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,441 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,441 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,441 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,441 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,441 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,441 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-10-27 23:53:05,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:05,441 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,441 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,441 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,442 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,441 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,442 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,442 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:05,442 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,442 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,442 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:05,442 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,443 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,443 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,443 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,543 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:05,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:05,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:05,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-10-27 23:53:05,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-10-27 23:53:05,562 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-10-27 23:53:05,562 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,562 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-10-27 23:53:05,562 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-10-27 23:53:05,563 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,563 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,569 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,569 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,569 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,569 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,570 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,570 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:05,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:05,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:05,571 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:05,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:05,571 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:05,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:05,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:05,571 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:05,572 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:05,571 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:05,572 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,572 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,572 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:05,572 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:05,572 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:05,572 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:05,572 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:05,572 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:05,572 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:05,572 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:05,572 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:05,572 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. started... 2023-10-27 23:53:05,572 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38d29e3b4a5d0b766af1d906cf63bcf5 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:53:05,582 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/b16cf9eb227d45c7b5bed322baca3e83 2023-10-27 23:53:05,582 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,591 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,591 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,591 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:05,591 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,591 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:05,595 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/b4ebf696c5494861bde6af773e7d1a36 2023-10-27 23:53:05,600 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/b16cf9eb227d45c7b5bed322baca3e83 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/b16cf9eb227d45c7b5bed322baca3e83 2023-10-27 23:53:05,605 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/b16cf9eb227d45c7b5bed322baca3e83, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:53:05,606 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/b4ebf696c5494861bde6af773e7d1a36 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/b4ebf696c5494861bde6af773e7d1a36 2023-10-27 23:53:05,612 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/b4ebf696c5494861bde6af773e7d1a36, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:53:05,613 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 38d29e3b4a5d0b766af1d906cf63bcf5 in 41ms, sequenceid=32, compaction requested=true 2023-10-27 23:53:05,613 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38d29e3b4a5d0b766af1d906cf63bcf5: 2023-10-27 23:53:05,613 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:05,613 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:05,613 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:05,613 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,613 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:05,613 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:05,624 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,624 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:05,625 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,625 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:05,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:05,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-10-27 23:53:05,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d24b8[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:05,627 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-10-27 23:53:05,627 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,636 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,636 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,636 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,636 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:05,636 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,637 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:05,637 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:05,637 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:05,637 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:05,637 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:05,637 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:05,637 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:05,637 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:05,637 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:05,644 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:05,644 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,644 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:05,653 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:05,653 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,653 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:05,653 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,653 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,653 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:05,653 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:53:05,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-10-27 23:53:05,656 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-10-27 23:53:05,656 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:05,656 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:05,656 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-10-27 23:53:05,656 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:05,666 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,666 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,666 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,666 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,666 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:05,666 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,667 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,666 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:05,667 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,667 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,669 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:05,669 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,669 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,698 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-10-27 23:53:05,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,699 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:05,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,699 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:05,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,750 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-27 23:53:05,799 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:05,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:05,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:05,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-10-27 23:53:05,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-10-27 23:53:05,822 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-10-27 23:53:05,822 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,822 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-10-27 23:53:05,822 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-10-27 23:53:05,823 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,823 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,836 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,836 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,836 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,836 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,837 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,837 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:05,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:05,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:05,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:05,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:05,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:05,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:05,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:05,838 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:05,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:05,838 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,838 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:05,838 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:05,838 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:05,838 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:05,838 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:05,838 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:05,838 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:05,838 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:05,838 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:05,838 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. started... 2023-10-27 23:53:05,838 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:05,838 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:05,838 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:05,839 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38d29e3b4a5d0b766af1d906cf63bcf5 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:53:05,839 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:05,839 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:05,848 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/d666e934306e4ae9989e6bd13f7515bd 2023-10-27 23:53:05,849 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,849 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,849 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,849 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,849 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:05,849 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:05,865 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/263ba16df33b4eaab6994bc169fb0207 2023-10-27 23:53:05,869 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/d666e934306e4ae9989e6bd13f7515bd as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/d666e934306e4ae9989e6bd13f7515bd 2023-10-27 23:53:05,875 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/d666e934306e4ae9989e6bd13f7515bd, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:53:05,876 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/263ba16df33b4eaab6994bc169fb0207 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/263ba16df33b4eaab6994bc169fb0207 2023-10-27 23:53:05,880 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/263ba16df33b4eaab6994bc169fb0207, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:53:05,881 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 38d29e3b4a5d0b766af1d906cf63bcf5 in 42ms, sequenceid=45, compaction requested=true 2023-10-27 23:53:05,881 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38d29e3b4a5d0b766af1d906cf63bcf5: 2023-10-27 23:53:05,882 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:05,882 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:05,882 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:05,882 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,882 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:05,882 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:05,891 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,891 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:05,891 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,892 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:05,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:05,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:05,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:05,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-10-27 23:53:05,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@482ff55f[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:05,894 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-10-27 23:53:05,894 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,903 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,903 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,903 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,903 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,903 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:05,903 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:05,903 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:05,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,903 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:05,903 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:05,903 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:05,903 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:05,903 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:05,903 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:05,903 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:05,911 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:05,911 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,911 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:05,911 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:05,919 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:05,919 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:05,919 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:05,920 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,920 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:05,919 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:05,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:53:05,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-10-27 23:53:05,922 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-10-27 23:53:05,923 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:05,923 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:05,923 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-10-27 23:53:05,923 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:05,933 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,933 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,933 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,933 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,933 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,933 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,933 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:05,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,933 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,934 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,934 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:05,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:05,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,936 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,936 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:05,936 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:05,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,944 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,944 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,944 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,944 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:05,944 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,944 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-10-27 23:53:05,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:05,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,944 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,945 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,945 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:05,945 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:05,945 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:05,945 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:05,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,945 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:05,945 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:05,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:05,946 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:05,946 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,946 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:05,946 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:05,946 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:05,946 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,045 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:06,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:06,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:06,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-10-27 23:53:06,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-10-27 23:53:06,066 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-10-27 23:53:06,066 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:06,066 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-10-27 23:53:06,066 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-10-27 23:53:06,066 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,066 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,074 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,074 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:06,074 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:06,074 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:06,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:06,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:06,075 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,075 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:06,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:06,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:06,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:06,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:06,076 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:06,076 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:06,076 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:06,076 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:06,076 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:06,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:06,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:06,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:06,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:06,079 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:06,079 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:06,079 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:06,079 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:06,079 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:06,079 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:06,079 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:06,079 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:06,081 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:06,081 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:06,081 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:06,081 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. started... 2023-10-27 23:53:06,082 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38d29e3b4a5d0b766af1d906cf63bcf5 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:53:06,086 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,092 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/9ba217caa46c4fde98f35d9ff2fe9449 2023-10-27 23:53:06,094 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,094 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,094 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:06,094 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,094 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:06,105 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/3f0e91b6001f4533a21b48c882d91924 2023-10-27 23:53:06,111 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/9ba217caa46c4fde98f35d9ff2fe9449 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/9ba217caa46c4fde98f35d9ff2fe9449 2023-10-27 23:53:06,116 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/9ba217caa46c4fde98f35d9ff2fe9449, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:53:06,117 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/3f0e91b6001f4533a21b48c882d91924 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/3f0e91b6001f4533a21b48c882d91924 2023-10-27 23:53:06,122 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/3f0e91b6001f4533a21b48c882d91924, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:53:06,123 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 38d29e3b4a5d0b766af1d906cf63bcf5 in 40ms, sequenceid=58, compaction requested=true 2023-10-27 23:53:06,123 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38d29e3b4a5d0b766af1d906cf63bcf5: 2023-10-27 23:53:06,123 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:06,123 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:06,123 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:06,123 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,123 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:06,123 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:06,133 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,133 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:06,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:06,133 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,133 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:06,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:06,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:06,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:06,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:06,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-10-27 23:53:06,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2456083c[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:06,135 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-10-27 23:53:06,136 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,144 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,144 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,144 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,145 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,145 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:06,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,145 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:06,145 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:06,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,145 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:06,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,145 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:06,145 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:06,145 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:06,145 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:06,145 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:06,145 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:06,174 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:06,174 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:06,174 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:06,186 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:06,186 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:06,186 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:06,194 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:06,194 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,194 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:06,195 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:53:06,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-10-27 23:53:06,198 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-10-27 23:53:06,198 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:06,198 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:06,198 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-10-27 23:53:06,198 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:06,208 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,208 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,208 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:06,208 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,208 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:06,208 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,208 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:06,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,208 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,208 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:06,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:06,209 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:06,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:06,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,210 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:06,210 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:06,211 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,219 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,219 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:06,219 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:06,219 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,219 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:06,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-10-27 23:53:06,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:06,219 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:06,220 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:06,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:06,220 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:06,220 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:06,220 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:06,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:06,220 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:06,220 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:06,220 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:06,220 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:06,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:06,220 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:06,221 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,221 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,221 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,221 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,221 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,321 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:06,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:06,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:06,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-10-27 23:53:06,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-10-27 23:53:06,342 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-10-27 23:53:06,343 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:06,343 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-10-27 23:53:06,343 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-10-27 23:53:06,343 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,343 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,353 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,353 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:06,353 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:06,353 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:06,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:06,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:06,353 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,353 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:06,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:06,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:06,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:06,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-10-27 23:53:06,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:06,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:06,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:06,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:06,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:06,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:06,355 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:06,355 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:06,355 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. started... 2023-10-27 23:53:06,356 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38d29e3b4a5d0b766af1d906cf63bcf5 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:53:06,361 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,365 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/7e8347c4e1484b5995f57459a42f05c2 2023-10-27 23:53:06,369 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,369 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,369 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:06,370 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,370 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:06,380 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/9ce57289c4c841909aa6d300fca592bc 2023-10-27 23:53:06,385 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/7e8347c4e1484b5995f57459a42f05c2 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/7e8347c4e1484b5995f57459a42f05c2 2023-10-27 23:53:06,390 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/7e8347c4e1484b5995f57459a42f05c2, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:53:06,391 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/9ce57289c4c841909aa6d300fca592bc as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/9ce57289c4c841909aa6d300fca592bc 2023-10-27 23:53:06,396 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/9ce57289c4c841909aa6d300fca592bc, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:53:06,397 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 38d29e3b4a5d0b766af1d906cf63bcf5 in 41ms, sequenceid=71, compaction requested=true 2023-10-27 23:53:06,397 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38d29e3b4a5d0b766af1d906cf63bcf5: 2023-10-27 23:53:06,397 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:06,397 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:06,397 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:06,397 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,397 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-10-27 23:53:06,397 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-10-27 23:53:06,408 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,408 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:06,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:06,408 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,408 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:06,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:06,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-10-27 23:53:06,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@142c1d53[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:06,410 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-10-27 23:53:06,410 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,419 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,419 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,419 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,420 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,420 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:06,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,420 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:06,420 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:06,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,420 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:06,420 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:06,420 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:06,420 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:06,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,420 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:53:06,420 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-10-27 23:53:06,420 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:06,428 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:06,428 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:06,428 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:06,436 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:06,436 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:06,436 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:06,444 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-10-27 23:53:06,444 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:06,445 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-10-27 23:53:06,444 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:06,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:06,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:06,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:06,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:06,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:53:06,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-10-27 23:53:06,448 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-10-27 23:53:06,448 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:06,448 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:06,448 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-10-27 23:53:06,448 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:06,458 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,458 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,458 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:06,458 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,458 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:06,458 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:06,458 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,458 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,459 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:06,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:06,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,459 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:06,461 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-10-27 23:53:06,461 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,461 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,483 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,483 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:06,483 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,483 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:06,483 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:06,483 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:06,483 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:06,483 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:06,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:06,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:06,483 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:06,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-10-27 23:53:06,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:06,483 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:06,484 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:06,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:06,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:06,484 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-10-27 23:53:06,484 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:06,484 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:06,484 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:06,484 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:06,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:06,484 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:06,484 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:06,484 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-10-27 23:53:06,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:06,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:06,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:06,485 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-10-27 23:53:06,584 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:06,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:06,847 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-10-27 23:53:06,848 DEBUG [Listener at localhost/43373] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:06,849 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:06,850 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:06,850 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:06,850 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:06,851 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:06,851 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:06,851 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:06,851 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:07,099 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 2 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,100 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:07,100 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:07,100 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:07,100 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-10-27 23:53:07,100 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:07,100 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:07,100 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:07,101 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:07,101 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:07,103 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-10-27 23:53:07,103 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:07,104 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:07,104 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:07,104 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:07,104 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:07,105 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:07,105 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:07,105 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:07,105 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:07,106 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:07,106 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:07,106 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:07,106 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:07,107 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,107 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:07,107 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,107 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,107 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,107 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 5 msec 2023-10-27 23:53:07,108 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:07,109 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:07,109 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450817109 2023-10-27 23:53:07,110 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:07,110 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,113 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:07,113 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:07,113 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:07,113 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450847113 2023-10-27 23:53:07,113 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450907113 2023-10-27 23:53:07,114 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:07,114 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:07,116 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:07,116 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,116 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:07,116 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:07,364 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 3 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,364 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:07,364 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:07,364 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:07,364 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:07,364 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:07,364 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:07,364 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:07,365 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:07,365 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:07,366 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-10-27 23:53:07,367 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:07,367 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:07,367 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:07,367 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:07,368 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:07,368 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:07,368 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:07,368 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:07,369 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:07,369 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:07,369 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:07,369 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:07,369 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:07,370 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,370 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:07,370 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,370 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,370 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,370 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 4 msec 2023-10-27 23:53:07,371 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:07,371 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:07,372 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450817372 2023-10-27 23:53:07,373 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,373 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:07,375 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:07,376 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:07,376 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:07,376 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450847376 2023-10-27 23:53:07,376 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450907376 2023-10-27 23:53:07,376 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:07,376 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:07,378 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN}] 2023-10-27 23:53:07,378 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:07,378 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,378 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:07,379 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:07,626 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,626 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:07,626 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:07,626 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:07,627 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:07,627 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:07,627 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:07,627 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:07,627 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:07,628 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:07,629 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-10-27 23:53:07,629 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:07,629 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:07,629 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:07,630 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:07,630 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:07,630 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:07,630 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:07,630 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:07,631 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:07,631 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:07,631 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:07,632 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:07,632 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:07,632 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,632 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:07,632 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,632 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,632 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,633 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 4 msec 2023-10-27 23:53:07,633 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:07,634 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:07,634 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450817634 2023-10-27 23:53:07,635 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:07,635 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,638 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:07,638 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:07,638 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:07,639 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450847639 2023-10-27 23:53:07,639 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450907639 2023-10-27 23:53:07,639 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:07,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN}] 2023-10-27 23:53:07,639 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:07,642 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:07,643 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:07,643 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450787643"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450787643"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450787643"}]},"ts":"1698450787643"} 2023-10-27 23:53:07,645 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:53:07,645 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:07,645 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:07,645 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:07,645 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:07,889 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,889 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:07,889 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:07,890 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:07,890 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:07,890 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:07,890 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:07,890 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:07,890 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:07,891 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:07,892 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-10-27 23:53:07,892 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-10-27 23:53:07,892 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:07,893 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:07,893 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:07,893 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:07,893 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:07,894 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:07,894 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:07,894 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:07,894 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:07,895 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:07,895 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:07,895 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:07,895 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:07,896 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,896 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:07,896 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:07,896 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,896 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,896 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:07,897 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:07,897 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:07,897 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 5 msec 2023-10-27 23:53:07,897 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:07,898 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:07,898 INFO [Listener at localhost/43373] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5 to restore RIT 2023-10-27 23:53:07,898 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450817898 2023-10-27 23:53:07,899 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:07,899 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:07,902 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:07,902 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSING, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:07,902 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:07,902 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450847902 2023-10-27 23:53:07,902 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450907902 2023-10-27 23:53:07,902 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:07,903 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:07,903 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450787903"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450787903"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450787903"}]},"ts":"1698450787903"} 2023-10-27 23:53:07,904 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:53:07,904 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:08,057 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:08,058 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,058 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 38d29e3b4a5d0b766af1d906cf63bcf5, disabling compactions & flushes 2023-10-27 23:53:08,058 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:08,058 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:08,058 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. after waiting 0 ms 2023-10-27 23:53:08,058 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:08,058 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 38d29e3b4a5d0b766af1d906cf63bcf5 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-10-27 23:53:08,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/4cbe2684635b4b9eb5f4336a72dc5e12 2023-10-27 23:53:08,081 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/c1b886e1dc7d452485339be6f02a3678 2023-10-27 23:53:08,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/4cbe2684635b4b9eb5f4336a72dc5e12 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4cbe2684635b4b9eb5f4336a72dc5e12 2023-10-27 23:53:08,094 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4cbe2684635b4b9eb5f4336a72dc5e12, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:53:08,095 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/c1b886e1dc7d452485339be6f02a3678 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/c1b886e1dc7d452485339be6f02a3678 2023-10-27 23:53:08,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/c1b886e1dc7d452485339be6f02a3678, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:53:08,101 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 38d29e3b4a5d0b766af1d906cf63bcf5 in 43ms, sequenceid=82, compaction requested=true 2023-10-27 23:53:08,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-10-27 23:53:08,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:53:08,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:08,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 38d29e3b4a5d0b766af1d906cf63bcf5: 2023-10-27 23:53:08,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,109 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED 2023-10-27 23:53:08,109 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450788109"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450788109"}]},"ts":"1698450788109"} 2023-10-27 23:53:08,110 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:08,110 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=46, ppid=45, state=SUCCESS, locked=true; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:08,110 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:08,110 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=46, ppid=45, state=SUCCESS, locked=true; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:08,153 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,153 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:08,153 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:08,153 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:08,154 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:08,154 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:08,154 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:08,154 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:08,154 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:08,155 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:08,156 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-10-27 23:53:08,156 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-10-27 23:53:08,156 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4] 2023-10-27 23:53:08,156 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:08,157 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:08,157 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:08,157 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:08,157 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:08,157 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:08,158 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:08,158 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:08,158 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:08,158 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:08,158 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:08,159 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:08,159 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:08,159 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,159 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:08,160 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:08,160 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:08,160 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,160 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,160 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,160 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:08,160 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:08,161 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:08,161 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 5 msec 2023-10-27 23:53:08,161 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:08,162 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:08,162 INFO [Listener at localhost/43373] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5 to restore RIT 2023-10-27 23:53:08,163 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450818163 2023-10-27 23:53:08,164 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:08,164 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:08,166 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:08,166 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:08,167 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:08,167 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450848167 2023-10-27 23:53:08,167 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450908167 2023-10-27 23:53:08,167 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:08,167 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:08,167 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450788167"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450788167"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450788167"}]},"ts":"1698450788167"} 2023-10-27 23:53:08,169 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:08,172 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=46, resume processing ppid=45 2023-10-27 23:53:08,172 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 in 265 msec 2023-10-27 23:53:08,172 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:08,172 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=45, ppid=44, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:08,172 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:08,172 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=45, ppid=44, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:08,417 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,417 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:08,417 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:08,418 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:08,418 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:08,418 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:08,418 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:08,418 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:08,418 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:08,419 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:08,420 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-10-27 23:53:08,420 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-10-27 23:53:08,420 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:08,420 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:08,421 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:08,421 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:08,421 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:08,421 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:08,421 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:08,422 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:08,422 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:08,422 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:08,422 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:08,422 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:08,422 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:08,423 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:08,423 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,424 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:08,424 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:08,424 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:08,424 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,424 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,424 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,424 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:08,424 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:08,424 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 5 msec 2023-10-27 23:53:08,425 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:08,425 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:08,425 INFO [Listener at localhost/43373] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5 to restore RIT 2023-10-27 23:53:08,426 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450818426 2023-10-27 23:53:08,427 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:08,427 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:08,429 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:08,430 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:08,430 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:08,430 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450848430 2023-10-27 23:53:08,430 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450908430 2023-10-27 23:53:08,430 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:08,430 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:08,432 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:08,437 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=45, resume processing ppid=44 2023-10-27 23:53:08,437 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN in 793 msec 2023-10-27 23:53:08,440 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:08,440 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,440 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:08,440 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:08,680 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 4 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,680 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:08,680 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:08,680 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:08,681 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:08,681 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:08,681 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:08,681 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:08,681 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:08,682 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:08,683 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-10-27 23:53:08,683 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:53:08,683 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:08,684 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:08,684 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:08,684 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:08,685 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:08,685 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:08,686 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:08,686 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:08,686 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:08,686 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:08,687 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:08,687 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:08,687 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:08,688 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:08,688 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,688 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:08,689 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:08,689 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:08,689 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:08,689 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:08,689 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 7 msec 2023-10-27 23:53:08,690 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:08,690 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:08,691 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450818691 2023-10-27 23:53:08,692 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:08,692 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,695 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:08,695 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:08,695 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:08,695 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450848695 2023-10-27 23:53:08,695 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450908695 2023-10-27 23:53:08,695 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:08,696 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,696 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,696 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:08,701 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:08,711 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=44 splitting 14 storefiles, region=38d29e3b4a5d0b766af1d906cf63bcf5, threads=14 2023-10-27 23:53:08,713 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/21df8aa0c3dd4184906a6001d2632643 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,713 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/22ccc037a33e47f0a14a23f59f65ec95 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,713 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/263ba16df33b4eaab6994bc169fb0207 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,713 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/3f0e91b6001f4533a21b48c882d91924 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,713 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/9ce57289c4c841909aa6d300fca592bc for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,713 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/b4ebf696c5494861bde6af773e7d1a36 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,713 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/c1b886e1dc7d452485339be6f02a3678 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,713 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/2defc9652d4341c6aad9dbeaac483c78 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,714 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4cbe2684635b4b9eb5f4336a72dc5e12 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,714 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/7e8347c4e1484b5995f57459a42f05c2 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,714 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/76f1e2d8901440ce89125dbf208fcfa0 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,714 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/9ba217caa46c4fde98f35d9ff2fe9449 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,714 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/b16cf9eb227d45c7b5bed322baca3e83 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,715 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/d666e934306e4ae9989e6bd13f7515bd for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,931 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/21df8aa0c3dd4184906a6001d2632643 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,934 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/c1b886e1dc7d452485339be6f02a3678 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,939 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/76f1e2d8901440ce89125dbf208fcfa0 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,943 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/b16cf9eb227d45c7b5bed322baca3e83 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,947 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/2defc9652d4341c6aad9dbeaac483c78 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,955 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/9ba217caa46c4fde98f35d9ff2fe9449 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,955 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/263ba16df33b4eaab6994bc169fb0207 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,967 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4cbe2684635b4b9eb5f4336a72dc5e12 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,986 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/3f0e91b6001f4533a21b48c882d91924 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,986 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/b4ebf696c5494861bde6af773e7d1a36 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,987 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/9ce57289c4c841909aa6d300fca592bc for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,989 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/7e8347c4e1484b5995f57459a42f05c2 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:08,991 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/22ccc037a33e47f0a14a23f59f65ec95 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,317 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/d666e934306e4ae9989e6bd13f7515bd for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,317 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=44 split storefiles for region 38d29e3b4a5d0b766af1d906cf63bcf5 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-10-27 23:53:09,359 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:09,359 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,359 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:09,359 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:09,446 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 5 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,446 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:09,446 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:09,446 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:09,448 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:09,448 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:09,448 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:09,449 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:09,449 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:09,449 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:09,450 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-10-27 23:53:09,451 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:53:09,451 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:09,451 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:09,452 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:09,452 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:09,452 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:09,452 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:09,453 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:09,453 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:09,453 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:09,453 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:09,453 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:09,453 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:09,454 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:09,454 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:09,454 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,455 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:09,455 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:09,455 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:09,455 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,455 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,455 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,455 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:09,455 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:09,455 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 5 msec 2023-10-27 23:53:09,456 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:09,457 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:09,457 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450819457 2023-10-27 23:53:09,458 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,458 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:09,460 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:09,461 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:09,461 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:09,461 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450849461 2023-10-27 23:53:09,461 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450909461 2023-10-27 23:53:09,461 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 2 msec 2023-10-27 23:53:09,470 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=44 splitting 14 storefiles, region=38d29e3b4a5d0b766af1d906cf63bcf5, threads=14 2023-10-27 23:53:09,471 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/21df8aa0c3dd4184906a6001d2632643 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,471 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/22ccc037a33e47f0a14a23f59f65ec95 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,471 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/263ba16df33b4eaab6994bc169fb0207 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,472 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/3f0e91b6001f4533a21b48c882d91924 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,472 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/9ce57289c4c841909aa6d300fca592bc for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,472 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/b4ebf696c5494861bde6af773e7d1a36 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,472 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/c1b886e1dc7d452485339be6f02a3678 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,472 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/2defc9652d4341c6aad9dbeaac483c78 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,472 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4cbe2684635b4b9eb5f4336a72dc5e12 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,473 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/76f1e2d8901440ce89125dbf208fcfa0 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,473 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/7e8347c4e1484b5995f57459a42f05c2 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,473 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/9ba217caa46c4fde98f35d9ff2fe9449 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,473 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/b16cf9eb227d45c7b5bed322baca3e83 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,473 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/d666e934306e4ae9989e6bd13f7515bd for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,658 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/22ccc037a33e47f0a14a23f59f65ec95 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,675 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/9ce57289c4c841909aa6d300fca592bc for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,675 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/21df8aa0c3dd4184906a6001d2632643 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,677 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/c1b886e1dc7d452485339be6f02a3678 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,677 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/3f0e91b6001f4533a21b48c882d91924 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,682 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/b16cf9eb227d45c7b5bed322baca3e83 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,691 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/d666e934306e4ae9989e6bd13f7515bd for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,702 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/76f1e2d8901440ce89125dbf208fcfa0 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,704 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/2defc9652d4341c6aad9dbeaac483c78 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,704 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/263ba16df33b4eaab6994bc169fb0207 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,712 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/7e8347c4e1484b5995f57459a42f05c2 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,719 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4cbe2684635b4b9eb5f4336a72dc5e12 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,728 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/b4ebf696c5494861bde6af773e7d1a36 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,732 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/9ba217caa46c4fde98f35d9ff2fe9449 for region: 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:09,733 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=44 split storefiles for region 38d29e3b4a5d0b766af1d906cf63bcf5 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-10-27 23:53:09,767 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:09,772 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/a03293152823ec15f91cc2b99007ad83/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-10-27 23:53:09,774 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/93f423a1d296125aea0b5218045bfe46/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-10-27 23:53:09,774 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:09,774 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,774 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:09,774 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:09,961 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(417): Restart 6 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,961 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:09,961 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:09,962 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:09,962 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:09,962 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:09,962 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:09,962 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:09,962 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:09,963 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:09,964 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-10-27 23:53:09,964 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:53:09,964 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:09,964 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:09,964 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:09,965 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:09,965 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:09,965 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:09,965 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:09,965 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:09,965 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:09,966 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:09,966 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:09,966 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:09,966 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:09,966 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:09,967 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,967 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:09,967 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:09,969 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:09,969 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,969 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,969 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,969 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:09,969 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:09,969 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 6 msec 2023-10-27 23:53:09,970 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:09,970 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:09,971 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450819971 2023-10-27 23:53:09,972 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:09,972 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,974 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:09,975 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:09,975 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:09,975 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450849975 2023-10-27 23:53:09,975 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450909975 2023-10-27 23:53:09,975 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 2 msec 2023-10-27 23:53:09,978 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:09,981 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:09,982 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:09,982 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:09,982 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:10,225 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(509): Restart 0 rollback state: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,226 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:10,226 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:10,226 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:10,226 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:10,226 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:10,226 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:10,226 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:10,226 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:10,227 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:10,228 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-10-27 23:53:10,228 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:53:10,228 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:10,228 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:10,228 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:10,228 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:10,228 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:10,229 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:10,229 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:10,229 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:10,229 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:10,229 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:10,230 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:10,230 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:10,230 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:10,230 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:10,230 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,230 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:10,231 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:10,231 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:10,231 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,231 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,231 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,231 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:10,231 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:10,231 DEBUG [Listener at localhost/43373] procedure2.StateMachineProcedure(225): Abort requested for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,231 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 4 msec 2023-10-27 23:53:10,232 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:10,232 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:10,233 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450820233 2023-10-27 23:53:10,233 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:10,236 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,238 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:10,238 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:10,238 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:10,238 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450850238 2023-10-27 23:53:10,238 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450910238 2023-10-27 23:53:10,238 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 4 msec 2023-10-27 23:53:10,239 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:10,250 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:10,250 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1580): TESTING: Kill before store update 2023-10-27 23:53:10,250 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:10,250 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1582) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1497) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1348) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:10,490 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(509): Restart 1 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,490 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:10,490 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:10,490 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:10,491 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:10,491 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:10,491 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:10,491 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:10,491 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:10,492 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:10,493 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10] 2023-10-27 23:53:10,493 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:53:10,493 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:10,494 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:10,494 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:10,494 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:10,494 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:10,495 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:10,495 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:10,495 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:10,495 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:10,495 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:10,496 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:10,496 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:10,496 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:10,496 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:10,497 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,497 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:10,497 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:10,497 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:10,497 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,497 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,497 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,497 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:10,497 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:10,497 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 5 msec 2023-10-27 23:53:10,498 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:10,498 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:10,499 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450820499 2023-10-27 23:53:10,506 INFO [PEWorker-1] procedure2.ProcedureExecutor(1356): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,506 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:10,508 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:10,508 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:10,508 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:10,508 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450850508 2023-10-27 23:53:10,508 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450910508 2023-10-27 23:53:10,508 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 2 msec 2023-10-27 23:53:10,508 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:10,511 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:10,511 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1580): TESTING: Kill before store update 2023-10-27 23:53:10,511 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:10,511 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1582) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1497) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1348) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:10,759 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(509): Restart 2 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,759 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:10,760 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:10,760 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:10,760 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:10,760 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:10,760 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:10,760 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:10,760 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:10,761 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:10,762 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-10-27 23:53:10,762 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:53:10,762 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:10,763 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:10,764 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:10,764 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:10,764 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:10,764 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:10,765 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:10,765 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:10,765 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:10,765 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:10,765 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:10,766 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:10,766 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:10,766 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:10,767 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,767 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:10,767 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:10,767 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:10,767 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,768 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,768 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,768 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:10,768 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:10,768 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 6 msec 2023-10-27 23:53:10,769 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:10,769 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:10,770 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450820770 2023-10-27 23:53:10,771 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:10,771 INFO [PEWorker-1] procedure2.ProcedureExecutor(1356): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:10,774 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:10,774 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:10,774 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:10,774 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450850774 2023-10-27 23:53:10,774 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450910774 2023-10-27 23:53:10,774 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:10,774 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:10,777 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:10,777 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1580): TESTING: Kill before store update 2023-10-27 23:53:10,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:10,777 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1582) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1497) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1348) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:11,024 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(509): Restart 3 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,025 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:11,025 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:11,025 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:11,026 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:11,026 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:11,026 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:11,026 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:11,026 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:11,027 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:11,028 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-10-27 23:53:11,028 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:53:11,028 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:11,028 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:11,029 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:11,029 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:11,029 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:11,029 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:11,030 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:11,030 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:11,030 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:11,030 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:11,031 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:11,031 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:11,031 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:11,031 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:11,032 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,032 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:11,032 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:11,032 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:11,032 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,033 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,033 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,033 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:11,033 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:11,033 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 5 msec 2023-10-27 23:53:11,034 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:11,034 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:11,034 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450821034 2023-10-27 23:53:11,036 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:11,036 INFO [PEWorker-1] procedure2.ProcedureExecutor(1356): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,039 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:11,039 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:11,039 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:11,039 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450851039 2023-10-27 23:53:11,039 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450911039 2023-10-27 23:53:11,039 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:11,040 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:11,044 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1053): Stored [pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN] 2023-10-27 23:53:11,044 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:11,044 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1580): TESTING: Kill before store update 2023-10-27 23:53:11,044 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:11,044 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1582) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1497) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1348) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:11,290 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(509): Restart 4 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,290 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:11,290 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:11,290 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:11,290 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:11,290 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:11,290 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:11,291 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:11,291 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:11,291 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:11,292 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-10-27 23:53:11,292 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:53:11,292 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:53:11,292 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-10-27 23:53:11,292 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:11,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:11,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:11,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:11,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:11,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:11,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:11,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:11,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:11,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:11,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:11,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:11,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:11,295 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,295 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:11,295 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:11,295 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:11,295 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:11,296 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,296 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,296 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,296 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:11,296 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:11,296 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:11,296 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 4 msec 2023-10-27 23:53:11,297 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:11,297 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:11,297 INFO [Listener at localhost/43373] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5 to restore RIT 2023-10-27 23:53:11,298 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450821298 2023-10-27 23:53:11,299 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:11,300 INFO [PEWorker-1] procedure2.ProcedureExecutor(1356): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,301 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:11,302 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:11,303 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:11,303 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:11,303 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450851303 2023-10-27 23:53:11,303 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450911303 2023-10-27 23:53:11,303 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 4 msec 2023-10-27 23:53:11,303 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:11,308 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:11,309 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1580): TESTING: Kill before store update 2023-10-27 23:53:11,309 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:11,309 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1582) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1497) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1348) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:11,553 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(509): Restart 5 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,553 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:11,553 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:11,553 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:11,553 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:11,554 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:11,554 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:11,554 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:11,554 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:11,554 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:11,555 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-10-27 23:53:11,555 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-10-27 23:53:11,555 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:11,555 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:11,556 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:11,556 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:11,556 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:11,556 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:11,556 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:11,556 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:11,556 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:11,557 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:11,557 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:11,557 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:11,557 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:11,557 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,557 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:11,557 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:11,557 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,558 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,558 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,558 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:11,558 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 3 msec 2023-10-27 23:53:11,559 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:11,559 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:11,559 INFO [Listener at localhost/43373] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5 to restore RIT 2023-10-27 23:53:11,559 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450821559 2023-10-27 23:53:11,561 INFO [PEWorker-1] procedure2.ProcedureExecutor(1356): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,561 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:11,562 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:11,564 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:11,564 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:11,564 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:11,564 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450851564 2023-10-27 23:53:11,564 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450911564 2023-10-27 23:53:11,564 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:11,564 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:11,567 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:11,567 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1580): TESTING: Kill before store update 2023-10-27 23:53:11,567 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:11,567 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1582) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1497) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1348) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:11,814 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(509): Restart 6 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,815 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:11,815 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:11,815 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:11,815 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:11,815 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:11,815 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:11,815 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:11,815 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:11,816 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:11,817 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-10-27 23:53:11,817 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-10-27 23:53:11,817 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:11,817 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:11,817 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:11,817 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:11,818 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:11,818 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:11,818 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:11,818 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:11,819 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:11,819 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:11,819 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:11,819 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:11,819 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:11,819 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,820 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:11,820 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:11,820 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,820 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,820 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,820 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:11,820 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 3 msec 2023-10-27 23:53:11,821 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:11,821 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:11,822 INFO [Listener at localhost/43373] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5 to restore RIT 2023-10-27 23:53:11,822 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450821822 2023-10-27 23:53:11,823 INFO [PEWorker-1] procedure2.ProcedureExecutor(1356): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:11,823 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:11,824 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:11,826 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:11,826 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:11,826 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:11,826 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450851826 2023-10-27 23:53:11,827 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450911827 2023-10-27 23:53:11,827 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:11,827 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:11,829 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:11,829 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1580): TESTING: Kill before store update 2023-10-27 23:53:11,829 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:11,829 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1582) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1497) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1348) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:12,077 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(509): Restart 7 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:12,077 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:12,077 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:12,077 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:12,077 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:12,078 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:12,078 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:12,078 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:12,078 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:12,079 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:12,079 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-10-27 23:53:12,079 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-10-27 23:53:12,080 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:12,080 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:12,080 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:12,080 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:12,080 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:12,081 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:12,081 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:12,081 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:12,081 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:12,082 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:12,082 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:12,082 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:12,082 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:12,083 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:12,083 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:12,083 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:12,083 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:12,083 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:12,083 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:12,083 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:53:12,084 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 4 msec 2023-10-27 23:53:12,084 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:12,085 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:12,085 INFO [Listener at localhost/43373] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5 to restore RIT 2023-10-27 23:53:12,085 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450822085 2023-10-27 23:53:12,087 INFO [PEWorker-1] procedure2.ProcedureExecutor(1356): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:12,087 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:12,089 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:12,091 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:12,091 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:12,091 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:12,091 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450852091 2023-10-27 23:53:12,091 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450912091 2023-10-27 23:53:12,091 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 4 msec 2023-10-27 23:53:12,092 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:53:12,094 INFO [PEWorker-1] procedure2.ProcedureExecutor(1527): Rolled back pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 exec-time=5.2440 sec 2023-10-27 23:53:12,094 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:12,095 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,43641,1698450767195; forceNewPlan=false, retain=true 2023-10-27 23:53:12,245 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-10-27 23:53:12,247 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:12,247 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450792247"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450792247"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450792247"}]},"ts":"1698450792247"} 2023-10-27 23:53:12,249 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE; OpenRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:53:12,249 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:53:12,249 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=47, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:12,249 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:53:12,249 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=47, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:53:12,342 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:53:12,342 WARN [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(526): ProcedureExecutor not running, may have been stopped by pending procedure due to KillAndToggleBeforeStoreUpdate flag. 2023-10-27 23:53:12,342 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:53:12,342 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:53:12,342 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:53:12,342 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:53:12,342 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:53:12,342 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:53:12,342 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:53:12,342 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:53:12,343 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:53:12,345 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[0] 2023-10-27 23:53:12,345 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:53:12,345 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:53:12,345 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:53:12,346 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:53:12,346 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:12,346 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:53:12,347 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:12,347 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:53:12,347 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:12,347 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:53:12,348 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:12,348 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:53:12,348 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:12,348 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:53:12,349 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:12,349 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:53:12,349 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:12,349 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 5 msec 2023-10-27 23:53:12,350 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:53:12,350 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:53:12,350 INFO [Listener at localhost/43373] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5 to restore RIT 2023-10-27 23:53:12,351 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450822351 2023-10-27 23:53:12,352 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:53:12,352 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:53:12,354 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:53:12,354 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=OPENING, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:53:12,354 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:53:12,354 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450852354 2023-10-27 23:53:12,354 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450912354 2023-10-27 23:53:12,354 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:53:12,355 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:12,355 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450792354"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450792354"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450792354"}]},"ts":"1698450792354"} 2023-10-27 23:53:12,357 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE; OpenRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:53:12,509 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:12,512 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:12,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 38d29e3b4a5d0b766af1d906cf63bcf5, NAME => 'testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:53:12,513 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:53:12,513 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:12,513 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:12,513 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:12,513 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:12,514 INFO [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:12,515 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1 2023-10-27 23:53:12,515 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1 2023-10-27 23:53:12,516 INFO [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38d29e3b4a5d0b766af1d906cf63bcf5 columnFamilyName cf1 2023-10-27 23:53:12,524 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/2defc9652d4341c6aad9dbeaac483c78 2023-10-27 23:53:12,527 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4cbe2684635b4b9eb5f4336a72dc5e12 2023-10-27 23:53:12,531 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/76f1e2d8901440ce89125dbf208fcfa0 2023-10-27 23:53:12,534 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/7e8347c4e1484b5995f57459a42f05c2 2023-10-27 23:53:12,537 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/9ba217caa46c4fde98f35d9ff2fe9449 2023-10-27 23:53:12,540 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/b16cf9eb227d45c7b5bed322baca3e83 2023-10-27 23:53:12,543 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/d666e934306e4ae9989e6bd13f7515bd 2023-10-27 23:53:12,544 INFO [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(310): Store=38d29e3b4a5d0b766af1d906cf63bcf5/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:12,544 INFO [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:12,545 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2 2023-10-27 23:53:12,545 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2 2023-10-27 23:53:12,545 INFO [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38d29e3b4a5d0b766af1d906cf63bcf5 columnFamilyName cf2 2023-10-27 23:53:12,551 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/21df8aa0c3dd4184906a6001d2632643 2023-10-27 23:53:12,555 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/22ccc037a33e47f0a14a23f59f65ec95 2023-10-27 23:53:12,558 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/263ba16df33b4eaab6994bc169fb0207 2023-10-27 23:53:12,562 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/3f0e91b6001f4533a21b48c882d91924 2023-10-27 23:53:12,565 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/9ce57289c4c841909aa6d300fca592bc 2023-10-27 23:53:12,569 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/b4ebf696c5494861bde6af773e7d1a36 2023-10-27 23:53:12,573 DEBUG [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/c1b886e1dc7d452485339be6f02a3678 2023-10-27 23:53:12,573 INFO [StoreOpener-38d29e3b4a5d0b766af1d906cf63bcf5-1] regionserver.HStore(310): Store=38d29e3b4a5d0b766af1d906cf63bcf5/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:12,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:12,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:12,577 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(602): Cleaned up old failed split transaction detritus: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.splits 2023-10-27 23:53:12,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRollbackAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:53:12,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:12,579 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 38d29e3b4a5d0b766af1d906cf63bcf5; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75031251, jitterRate=0.11805276572704315}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:53:12,580 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 38d29e3b4a5d0b766af1d906cf63bcf5: 2023-10-27 23:53:12,580 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5., pid=48, masterSystemTime=1698450792509 2023-10-27 23:53:12,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:12,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:53:12,581 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:53:12,581 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:53:12,582 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-10-27 23:53:12,582 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-10-27 23:53:12,582 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1912): 38d29e3b4a5d0b766af1d906cf63bcf5/cf2 is initiating minor compaction (all files) 2023-10-27 23:53:12,583 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1912): 38d29e3b4a5d0b766af1d906cf63bcf5/cf1 is initiating minor compaction (all files) 2023-10-27 23:53:12,583 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38d29e3b4a5d0b766af1d906cf63bcf5/cf2 in testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:12,583 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 38d29e3b4a5d0b766af1d906cf63bcf5/cf1 in testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:12,583 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/2defc9652d4341c6aad9dbeaac483c78, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/76f1e2d8901440ce89125dbf208fcfa0, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/b16cf9eb227d45c7b5bed322baca3e83, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/d666e934306e4ae9989e6bd13f7515bd, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/9ba217caa46c4fde98f35d9ff2fe9449, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/7e8347c4e1484b5995f57459a42f05c2, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4cbe2684635b4b9eb5f4336a72dc5e12] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp, totalSize=34.9 K 2023-10-27 23:53:12,583 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/21df8aa0c3dd4184906a6001d2632643, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/22ccc037a33e47f0a14a23f59f65ec95, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/b4ebf696c5494861bde6af773e7d1a36, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/263ba16df33b4eaab6994bc169fb0207, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/3f0e91b6001f4533a21b48c882d91924, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/9ce57289c4c841909aa6d300fca592bc, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/c1b886e1dc7d452485339be6f02a3678] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp, totalSize=34.9 K 2023-10-27 23:53:12,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:12,584 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:12,584 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 21df8aa0c3dd4184906a6001d2632643, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450785043 2023-10-27 23:53:12,584 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 2defc9652d4341c6aad9dbeaac483c78, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450785043 2023-10-27 23:53:12,585 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:12,585 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450792584"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450792584"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450792584"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450792584"}]},"ts":"1698450792584"} 2023-10-27 23:53:12,585 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 22ccc037a33e47f0a14a23f59f65ec95, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450785273 2023-10-27 23:53:12,585 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 76f1e2d8901440ce89125dbf208fcfa0, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450785273 2023-10-27 23:53:12,585 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting b4ebf696c5494861bde6af773e7d1a36, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450785544 2023-10-27 23:53:12,585 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting b16cf9eb227d45c7b5bed322baca3e83, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450785544 2023-10-27 23:53:12,585 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 263ba16df33b4eaab6994bc169fb0207, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450785802 2023-10-27 23:53:12,586 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting d666e934306e4ae9989e6bd13f7515bd, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450785802 2023-10-27 23:53:12,586 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 3f0e91b6001f4533a21b48c882d91924, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450786047 2023-10-27 23:53:12,586 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 9ba217caa46c4fde98f35d9ff2fe9449, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450786047 2023-10-27 23:53:12,586 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 9ce57289c4c841909aa6d300fca592bc, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450786323 2023-10-27 23:53:12,586 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 7e8347c4e1484b5995f57459a42f05c2, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450786323 2023-10-27 23:53:12,587 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting c1b886e1dc7d452485339be6f02a3678, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1698450786586 2023-10-27 23:53:12,587 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 4cbe2684635b4b9eb5f4336a72dc5e12, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1698450786586 2023-10-27 23:53:12,593 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=48, resume processing ppid=47 2023-10-27 23:53:12,593 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=48, ppid=47, state=SUCCESS; OpenRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 in 230 msec 2023-10-27 23:53:12,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN in 1.5510 sec 2023-10-27 23:53:12,606 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(335): procId=44 exception: org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested 2023-10-27 23:53:12,611 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] throttle.PressureAwareThroughputController(145): 38d29e3b4a5d0b766af1d906cf63bcf5#cf2#compaction#50 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:12,615 DEBUG [Listener at localhost/43373] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testRollbackAndDoubleExecution get assigned. Timeout = 1500ms 2023-10-27 23:53:12,617 INFO [Listener at localhost/43373] hbase.Waiter(180): Waiting up to [1,500] milli-secs(wait.for.ratio=[1]) 2023-10-27 23:53:12,620 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] throttle.PressureAwareThroughputController(145): 38d29e3b4a5d0b766af1d906cf63bcf5#cf1#compaction#51 average throughput is 1.72 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:12,621 INFO [Listener at localhost/43373] hbase.HBaseTestingUtility(3484): All regions for table testRollbackAndDoubleExecution assigned to meta. Checking AM states. 2023-10-27 23:53:12,622 INFO [Listener at localhost/43373] hbase.Waiter(180): Waiting up to [1,500] milli-secs(wait.for.ratio=[1]) 2023-10-27 23:53:12,622 INFO [Listener at localhost/43373] hbase.HBaseTestingUtility(3504): All regions for table testRollbackAndDoubleExecution assigned. 2023-10-27 23:53:12,644 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf2/8bb12292b7da40d1b052524a1d16db6e as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/8bb12292b7da40d1b052524a1d16db6e 2023-10-27 23:53:12,649 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/.tmp/cf1/4d13c525b2d64d81a753efe7b7bca87f as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4d13c525b2d64d81a753efe7b7bca87f 2023-10-27 23:53:12,654 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 38d29e3b4a5d0b766af1d906cf63bcf5/cf2 of 38d29e3b4a5d0b766af1d906cf63bcf5 into 8bb12292b7da40d1b052524a1d16db6e(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:12,654 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38d29e3b4a5d0b766af1d906cf63bcf5: 2023-10-27 23:53:12,654 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5., storeName=38d29e3b4a5d0b766af1d906cf63bcf5/cf2, priority=9, startTime=1698450792581; duration=0sec 2023-10-27 23:53:12,655 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:12,656 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 38d29e3b4a5d0b766af1d906cf63bcf5/cf1 of 38d29e3b4a5d0b766af1d906cf63bcf5 into 4d13c525b2d64d81a753efe7b7bca87f(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:12,656 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38d29e3b4a5d0b766af1d906cf63bcf5: 2023-10-27 23:53:12,657 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5., storeName=38d29e3b4a5d0b766af1d906cf63bcf5/cf1, priority=9, startTime=1698450792580; duration=0sec 2023-10-27 23:53:12,657 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:12,666 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:53:12,667 INFO [Listener at localhost/43373] client.HBaseAdmin$15(890): Started disable of testRollbackAndDoubleExecution 2023-10-27 23:53:12,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable testRollbackAndDoubleExecution 2023-10-27 23:53:12,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:12,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=49 2023-10-27 23:53:12,671 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450792671"}]},"ts":"1698450792671"} 2023-10-27 23:53:12,672 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=DISABLING in hbase:meta 2023-10-27 23:53:12,702 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRollbackAndDoubleExecution to state=DISABLING 2023-10-27 23:53:12,703 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN}] 2023-10-27 23:53:12,704 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN 2023-10-27 23:53:12,705 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:12,705 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450792705"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450792705"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450792705"}]},"ts":"1698450792705"} 2023-10-27 23:53:12,706 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:53:12,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=49 2023-10-27 23:53:12,858 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:12,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:12,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 38d29e3b4a5d0b766af1d906cf63bcf5, disabling compactions & flushes 2023-10-27 23:53:12,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:12,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:12,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. after waiting 0 ms 2023-10-27 23:53:12,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:12,859 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/2defc9652d4341c6aad9dbeaac483c78, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/76f1e2d8901440ce89125dbf208fcfa0, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/b16cf9eb227d45c7b5bed322baca3e83, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/d666e934306e4ae9989e6bd13f7515bd, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/9ba217caa46c4fde98f35d9ff2fe9449, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/7e8347c4e1484b5995f57459a42f05c2, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4cbe2684635b4b9eb5f4336a72dc5e12] to archive 2023-10-27 23:53:12,860 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:53:12,862 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/2defc9652d4341c6aad9dbeaac483c78 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/2defc9652d4341c6aad9dbeaac483c78 2023-10-27 23:53:12,863 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/76f1e2d8901440ce89125dbf208fcfa0 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/76f1e2d8901440ce89125dbf208fcfa0 2023-10-27 23:53:12,864 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/b16cf9eb227d45c7b5bed322baca3e83 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/b16cf9eb227d45c7b5bed322baca3e83 2023-10-27 23:53:12,865 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/d666e934306e4ae9989e6bd13f7515bd to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/d666e934306e4ae9989e6bd13f7515bd 2023-10-27 23:53:12,867 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/9ba217caa46c4fde98f35d9ff2fe9449 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/9ba217caa46c4fde98f35d9ff2fe9449 2023-10-27 23:53:12,868 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/7e8347c4e1484b5995f57459a42f05c2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/7e8347c4e1484b5995f57459a42f05c2 2023-10-27 23:53:12,870 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4cbe2684635b4b9eb5f4336a72dc5e12 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4cbe2684635b4b9eb5f4336a72dc5e12 2023-10-27 23:53:12,874 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/21df8aa0c3dd4184906a6001d2632643, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/22ccc037a33e47f0a14a23f59f65ec95, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/b4ebf696c5494861bde6af773e7d1a36, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/263ba16df33b4eaab6994bc169fb0207, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/3f0e91b6001f4533a21b48c882d91924, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/9ce57289c4c841909aa6d300fca592bc, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/c1b886e1dc7d452485339be6f02a3678] to archive 2023-10-27 23:53:12,875 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:53:12,877 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/21df8aa0c3dd4184906a6001d2632643 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/21df8aa0c3dd4184906a6001d2632643 2023-10-27 23:53:12,878 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/22ccc037a33e47f0a14a23f59f65ec95 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/22ccc037a33e47f0a14a23f59f65ec95 2023-10-27 23:53:12,879 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/b4ebf696c5494861bde6af773e7d1a36 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/b4ebf696c5494861bde6af773e7d1a36 2023-10-27 23:53:12,880 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/263ba16df33b4eaab6994bc169fb0207 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/263ba16df33b4eaab6994bc169fb0207 2023-10-27 23:53:12,881 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/3f0e91b6001f4533a21b48c882d91924 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/3f0e91b6001f4533a21b48c882d91924 2023-10-27 23:53:12,882 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/9ce57289c4c841909aa6d300fca592bc to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/9ce57289c4c841909aa6d300fca592bc 2023-10-27 23:53:12,883 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/c1b886e1dc7d452485339be6f02a3678 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/c1b886e1dc7d452485339be6f02a3678 2023-10-27 23:53:12,887 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-10-27 23:53:12,888 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:53:12,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5. 2023-10-27 23:53:12,888 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 38d29e3b4a5d0b766af1d906cf63bcf5: 2023-10-27 23:53:12,889 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:12,890 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=38d29e3b4a5d0b766af1d906cf63bcf5, regionState=CLOSED 2023-10-27 23:53:12,890 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450792890"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450792890"}]},"ts":"1698450792890"} 2023-10-27 23:53:12,892 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=51, resume processing ppid=50 2023-10-27 23:53:12,892 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=51, ppid=50, state=SUCCESS; CloseRegionProcedure 38d29e3b4a5d0b766af1d906cf63bcf5, server=jenkins-hbase16.apache.org,43641,1698450767195 in 185 msec 2023-10-27 23:53:12,894 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=50, resume processing ppid=49 2023-10-27 23:53:12,894 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, UNASSIGN in 189 msec 2023-10-27 23:53:12,894 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450792894"}]},"ts":"1698450792894"} 2023-10-27 23:53:12,896 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=DISABLED in hbase:meta 2023-10-27 23:53:12,902 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRollbackAndDoubleExecution to state=DISABLED 2023-10-27 23:53:12,904 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution in 235 msec 2023-10-27 23:53:12,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=49 2023-10-27 23:53:12,974 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRollbackAndDoubleExecution, procId: 49 completed 2023-10-27 23:53:12,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete testRollbackAndDoubleExecution 2023-10-27 23:53:12,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=52, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:12,977 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=52, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:12,978 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=52, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:12,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=52 2023-10-27 23:53:12,981 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:12,983 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/recovered.edits] 2023-10-27 23:53:12,986 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4d13c525b2d64d81a753efe7b7bca87f to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf1/4d13c525b2d64d81a753efe7b7bca87f 2023-10-27 23:53:12,988 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/8bb12292b7da40d1b052524a1d16db6e to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/cf2/8bb12292b7da40d1b052524a1d16db6e 2023-10-27 23:53:12,991 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/recovered.edits/90.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5/recovered.edits/90.seqid 2023-10-27 23:53:12,992 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackAndDoubleExecution/38d29e3b4a5d0b766af1d906cf63bcf5 2023-10-27 23:53:12,992 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackAndDoubleExecution regions 2023-10-27 23:53:12,995 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=52, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:12,997 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testRollbackAndDoubleExecution from hbase:meta 2023-10-27 23:53:12,998 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRollbackAndDoubleExecution' descriptor. 2023-10-27 23:53:13,000 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=52, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:13,000 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRollbackAndDoubleExecution' from region states. 2023-10-27 23:53:13,000 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450793000"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:13,001 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-10-27 23:53:13,001 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 38d29e3b4a5d0b766af1d906cf63bcf5, NAME => 'testRollbackAndDoubleExecution,,1698450784534.38d29e3b4a5d0b766af1d906cf63bcf5.', STARTKEY => '', ENDKEY => ''}] 2023-10-27 23:53:13,001 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRollbackAndDoubleExecution' as deleted. 2023-10-27 23:53:13,002 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1698450793001"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:13,003 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRollbackAndDoubleExecution state from META 2023-10-27 23:53:13,011 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=52, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:53:13,012 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution in 36 msec 2023-10-27 23:53:13,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=52 2023-10-27 23:53:13,080 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRollbackAndDoubleExecution, procId: 52 completed 2023-10-27 23:53:13,093 INFO [Listener at localhost/43373] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRollbackAndDoubleExecution Thread=401 (was 397) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:55048 [Waiting for operation #34] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:37342 [Waiting for operation #25] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:43588 [Waiting for operation #15] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:43610 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:43714 [Waiting for operation #30] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:37356 [Waiting for operation #28] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:33920 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:57432 [Waiting for operation #33] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:316) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:373) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:37166 [Waiting for operation #31] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:316) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:373) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:43548 [Waiting for operation #28] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:44140 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:55076 [Waiting for operation #41] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:43698 [Waiting for operation #23] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:1;jenkins-hbase16:43641-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:37924 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:55046 [Waiting for operation #36] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_920238258_17 at /127.0.0.1:44476 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=865 (was 844) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=348 (was 376), ProcessCount=171 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=5526 (was 5667) 2023-10-27 23:53:13,105 INFO [Listener at localhost/43373] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionNoStoreFile Thread=401, OpenFileDescriptor=865, MaxFileDescriptor=60000, SystemLoadAverage=348, ProcessCount=171, AvailableMemoryMB=5525 2023-10-27 23:53:13,105 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:53:13,116 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-10-27 23:53:13,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(492): Client=jenkins//188.40.62.62 set balanceSwitch=false 2023-10-27 23:53:13,118 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=53, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:53:13,120 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_PRE_OPERATION 2023-10-27 23:53:13,120 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-10-27 23:53:13,122 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:13,123 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15 empty. 2023-10-27 23:53:13,123 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:13,123 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionNoStoreFile regions 2023-10-27 23:53:13,132 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/.tabledesc/.tableinfo.0000000001 2023-10-27 23:53:13,133 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(7675): creating {ENCODED => aa542793a0b70c04a27ad5d5b58f9b15, NAME => 'testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionNoStoreFile', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp 2023-10-27 23:53:13,140 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:13,141 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1604): Closing aa542793a0b70c04a27ad5d5b58f9b15, disabling compactions & flushes 2023-10-27 23:53:13,141 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15. 2023-10-27 23:53:13,141 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15. 2023-10-27 23:53:13,141 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15. after waiting 0 ms 2023-10-27 23:53:13,141 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15. 2023-10-27 23:53:13,141 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15. 2023-10-27 23:53:13,141 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1558): Region close journal for aa542793a0b70c04a27ad5d5b58f9b15: 2023-10-27 23:53:13,143 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_ADD_TO_META 2023-10-27 23:53:13,144 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1698450793144"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450793144"}]},"ts":"1698450793144"} 2023-10-27 23:53:13,147 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-10-27 23:53:13,148 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-10-27 23:53:13,148 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450793148"}]},"ts":"1698450793148"} 2023-10-27 23:53:13,149 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=ENABLING in hbase:meta 2023-10-27 23:53:13,169 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-10-27 23:53:13,170 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-27 23:53:13,170 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-10-27 23:53:13,170 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-10-27 23:53:13,170 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-10-27 23:53:13,171 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=aa542793a0b70c04a27ad5d5b58f9b15, ASSIGN}] 2023-10-27 23:53:13,172 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=aa542793a0b70c04a27ad5d5b58f9b15, ASSIGN 2023-10-27 23:53:13,173 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=aa542793a0b70c04a27ad5d5b58f9b15, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,39059,1698450767147; forceNewPlan=false, retain=false 2023-10-27 23:53:13,323 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-10-27 23:53:13,324 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=aa542793a0b70c04a27ad5d5b58f9b15, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:13,324 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1698450793324"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450793324"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450793324"}]},"ts":"1698450793324"} 2023-10-27 23:53:13,326 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE; OpenRegionProcedure aa542793a0b70c04a27ad5d5b58f9b15, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:13,478 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:13,484 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15. 2023-10-27 23:53:13,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => aa542793a0b70c04a27ad5d5b58f9b15, NAME => 'testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:53:13,484 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:53:13,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:13,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:13,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:13,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:13,486 INFO [StoreOpener-aa542793a0b70c04a27ad5d5b58f9b15-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:13,487 DEBUG [StoreOpener-aa542793a0b70c04a27ad5d5b58f9b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15/cf1 2023-10-27 23:53:13,488 DEBUG [StoreOpener-aa542793a0b70c04a27ad5d5b58f9b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15/cf1 2023-10-27 23:53:13,488 INFO [StoreOpener-aa542793a0b70c04a27ad5d5b58f9b15-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aa542793a0b70c04a27ad5d5b58f9b15 columnFamilyName cf1 2023-10-27 23:53:13,489 INFO [StoreOpener-aa542793a0b70c04a27ad5d5b58f9b15-1] regionserver.HStore(310): Store=aa542793a0b70c04a27ad5d5b58f9b15/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:13,489 INFO [StoreOpener-aa542793a0b70c04a27ad5d5b58f9b15-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:13,490 DEBUG [StoreOpener-aa542793a0b70c04a27ad5d5b58f9b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15/cf2 2023-10-27 23:53:13,491 DEBUG [StoreOpener-aa542793a0b70c04a27ad5d5b58f9b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15/cf2 2023-10-27 23:53:13,491 INFO [StoreOpener-aa542793a0b70c04a27ad5d5b58f9b15-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aa542793a0b70c04a27ad5d5b58f9b15 columnFamilyName cf2 2023-10-27 23:53:13,492 INFO [StoreOpener-aa542793a0b70c04a27ad5d5b58f9b15-1] regionserver.HStore(310): Store=aa542793a0b70c04a27ad5d5b58f9b15/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:13,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:13,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:13,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:53:13,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:13,499 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-10-27 23:53:13,500 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened aa542793a0b70c04a27ad5d5b58f9b15; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71041684, jitterRate=0.05860358476638794}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:53:13,500 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for aa542793a0b70c04a27ad5d5b58f9b15: 2023-10-27 23:53:13,500 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15., pid=55, masterSystemTime=1698450793478 2023-10-27 23:53:13,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15. 2023-10-27 23:53:13,502 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15. 2023-10-27 23:53:13,502 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=aa542793a0b70c04a27ad5d5b58f9b15, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:13,502 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1698450793502"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450793502"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450793502"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450793502"}]},"ts":"1698450793502"} 2023-10-27 23:53:13,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=55, resume processing ppid=54 2023-10-27 23:53:13,506 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=55, ppid=54, state=SUCCESS; OpenRegionProcedure aa542793a0b70c04a27ad5d5b58f9b15, server=jenkins-hbase16.apache.org,39059,1698450767147 in 178 msec 2023-10-27 23:53:13,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=54, resume processing ppid=53 2023-10-27 23:53:13,508 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=54, ppid=53, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=aa542793a0b70c04a27ad5d5b58f9b15, ASSIGN in 335 msec 2023-10-27 23:53:13,508 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-10-27 23:53:13,508 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450793508"}]},"ts":"1698450793508"} 2023-10-27 23:53:13,510 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=ENABLED in hbase:meta 2023-10-27 23:53:13,542 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_POST_OPERATION 2023-10-27 23:53:13,543 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile in 425 msec 2023-10-27 23:53:13,608 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-27 23:53:13,619 DEBUG [Listener at localhost/43373] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:13,620 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:13,621 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:13,621 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:13,621 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:13,623 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=57, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=aa542793a0b70c04a27ad5d5b58f9b15, UNASSIGN}] 2023-10-27 23:53:13,624 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=57, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=aa542793a0b70c04a27ad5d5b58f9b15, UNASSIGN 2023-10-27 23:53:13,625 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=aa542793a0b70c04a27ad5d5b58f9b15, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:13,625 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1698450793625"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450793625"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450793625"}]},"ts":"1698450793625"} 2023-10-27 23:53:13,627 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE; CloseRegionProcedure aa542793a0b70c04a27ad5d5b58f9b15, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:13,778 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:13,779 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:13,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing aa542793a0b70c04a27ad5d5b58f9b15, disabling compactions & flushes 2023-10-27 23:53:13,780 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15. 2023-10-27 23:53:13,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15. 2023-10-27 23:53:13,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15. after waiting 0 ms 2023-10-27 23:53:13,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15. 2023-10-27 23:53:13,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-10-27 23:53:13,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:53:13,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15. 2023-10-27 23:53:13,786 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for aa542793a0b70c04a27ad5d5b58f9b15: 2023-10-27 23:53:13,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:13,788 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=aa542793a0b70c04a27ad5d5b58f9b15, regionState=CLOSED 2023-10-27 23:53:13,788 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1698450793788"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450793788"}]},"ts":"1698450793788"} 2023-10-27 23:53:13,791 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=58, resume processing ppid=57 2023-10-27 23:53:13,791 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=58, ppid=57, state=SUCCESS; CloseRegionProcedure aa542793a0b70c04a27ad5d5b58f9b15, server=jenkins-hbase16.apache.org,39059,1698450767147 in 163 msec 2023-10-27 23:53:13,793 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=57, resume processing ppid=56 2023-10-27 23:53:13,793 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=57, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=aa542793a0b70c04a27ad5d5b58f9b15, UNASSIGN in 168 msec 2023-10-27 23:53:13,816 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=-1 2023-10-27 23:53:13,818 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=-1 2023-10-27 23:53:13,820 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1698450793820"},{"qualifier":"splitA","vlen":67,"tag":[],"timestamp":"1698450793820"},{"qualifier":"splitB","vlen":67,"tag":[],"timestamp":"1698450793820"}]},"ts":"1698450793820"} 2023-10-27 23:53:13,820 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450793820"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450793820"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450793820"}]},"ts":"1698450793820"} 2023-10-27 23:53:13,820 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450793820"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450793820"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450793820"}]},"ts":"1698450793820"} 2023-10-27 23:53:13,823 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=c6d17e2577f36e5cbf594fca675240be, ASSIGN}, {pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=433bacad157e4d0e8f716e1942033cb2, ASSIGN}] 2023-10-27 23:53:13,824 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=433bacad157e4d0e8f716e1942033cb2, ASSIGN 2023-10-27 23:53:13,825 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=433bacad157e4d0e8f716e1942033cb2, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase16.apache.org,39059,1698450767147; forceNewPlan=false, retain=false 2023-10-27 23:53:13,825 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=c6d17e2577f36e5cbf594fca675240be, ASSIGN 2023-10-27 23:53:13,826 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=c6d17e2577f36e5cbf594fca675240be, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase16.apache.org,39059,1698450767147; forceNewPlan=false, retain=false 2023-10-27 23:53:13,975 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-10-27 23:53:13,976 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=60 updating hbase:meta row=433bacad157e4d0e8f716e1942033cb2, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:13,977 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450793976"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450793976"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450793976"}]},"ts":"1698450793976"} 2023-10-27 23:53:13,978 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=61, ppid=60, state=RUNNABLE; OpenRegionProcedure 433bacad157e4d0e8f716e1942033cb2, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:13,981 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=c6d17e2577f36e5cbf594fca675240be, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:13,981 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450793981"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450793981"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450793981"}]},"ts":"1698450793981"} 2023-10-27 23:53:13,982 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=62, ppid=59, state=RUNNABLE; OpenRegionProcedure c6d17e2577f36e5cbf594fca675240be, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:14,130 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:14,133 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be. 2023-10-27 23:53:14,133 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c6d17e2577f36e5cbf594fca675240be, NAME => 'testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be.', STARTKEY => '', ENDKEY => '41'} 2023-10-27 23:53:14,134 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:53:14,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile c6d17e2577f36e5cbf594fca675240be 2023-10-27 23:53:14,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:14,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for c6d17e2577f36e5cbf594fca675240be 2023-10-27 23:53:14,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for c6d17e2577f36e5cbf594fca675240be 2023-10-27 23:53:14,135 INFO [StoreOpener-c6d17e2577f36e5cbf594fca675240be-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region c6d17e2577f36e5cbf594fca675240be 2023-10-27 23:53:14,137 DEBUG [StoreOpener-c6d17e2577f36e5cbf594fca675240be-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be/cf1 2023-10-27 23:53:14,137 DEBUG [StoreOpener-c6d17e2577f36e5cbf594fca675240be-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be/cf1 2023-10-27 23:53:14,137 INFO [StoreOpener-c6d17e2577f36e5cbf594fca675240be-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6d17e2577f36e5cbf594fca675240be columnFamilyName cf1 2023-10-27 23:53:14,138 INFO [StoreOpener-c6d17e2577f36e5cbf594fca675240be-1] regionserver.HStore(310): Store=c6d17e2577f36e5cbf594fca675240be/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:14,138 INFO [StoreOpener-c6d17e2577f36e5cbf594fca675240be-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region c6d17e2577f36e5cbf594fca675240be 2023-10-27 23:53:14,139 DEBUG [StoreOpener-c6d17e2577f36e5cbf594fca675240be-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be/cf2 2023-10-27 23:53:14,140 DEBUG [StoreOpener-c6d17e2577f36e5cbf594fca675240be-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be/cf2 2023-10-27 23:53:14,140 INFO [StoreOpener-c6d17e2577f36e5cbf594fca675240be-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6d17e2577f36e5cbf594fca675240be columnFamilyName cf2 2023-10-27 23:53:14,141 INFO [StoreOpener-c6d17e2577f36e5cbf594fca675240be-1] regionserver.HStore(310): Store=c6d17e2577f36e5cbf594fca675240be/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:14,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be 2023-10-27 23:53:14,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be 2023-10-27 23:53:14,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:53:14,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for c6d17e2577f36e5cbf594fca675240be 2023-10-27 23:53:14,146 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened c6d17e2577f36e5cbf594fca675240be; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73648279, jitterRate=0.09744487702846527}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:53:14,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for c6d17e2577f36e5cbf594fca675240be: 2023-10-27 23:53:14,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be., pid=62, masterSystemTime=1698450794130 2023-10-27 23:53:14,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be. 2023-10-27 23:53:14,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be. 2023-10-27 23:53:14,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2. 2023-10-27 23:53:14,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 433bacad157e4d0e8f716e1942033cb2, NAME => 'testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2.', STARTKEY => '41', ENDKEY => ''} 2023-10-27 23:53:14,148 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=c6d17e2577f36e5cbf594fca675240be, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:14,149 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450794148"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450794148"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450794148"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450794148"}]},"ts":"1698450794148"} 2023-10-27 23:53:14,149 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:53:14,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile 433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:14,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:14,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:14,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:14,150 INFO [StoreOpener-433bacad157e4d0e8f716e1942033cb2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:14,152 DEBUG [StoreOpener-433bacad157e4d0e8f716e1942033cb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2/cf1 2023-10-27 23:53:14,152 DEBUG [StoreOpener-433bacad157e4d0e8f716e1942033cb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2/cf1 2023-10-27 23:53:14,152 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=62, resume processing ppid=59 2023-10-27 23:53:14,152 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=62, ppid=59, state=SUCCESS; OpenRegionProcedure c6d17e2577f36e5cbf594fca675240be, server=jenkins-hbase16.apache.org,39059,1698450767147 in 168 msec 2023-10-27 23:53:14,152 INFO [StoreOpener-433bacad157e4d0e8f716e1942033cb2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 433bacad157e4d0e8f716e1942033cb2 columnFamilyName cf1 2023-10-27 23:53:14,153 INFO [StoreOpener-433bacad157e4d0e8f716e1942033cb2-1] regionserver.HStore(310): Store=433bacad157e4d0e8f716e1942033cb2/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:14,153 INFO [StoreOpener-433bacad157e4d0e8f716e1942033cb2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:14,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=59, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=c6d17e2577f36e5cbf594fca675240be, ASSIGN in 329 msec 2023-10-27 23:53:14,154 DEBUG [StoreOpener-433bacad157e4d0e8f716e1942033cb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2/cf2 2023-10-27 23:53:14,154 DEBUG [StoreOpener-433bacad157e4d0e8f716e1942033cb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2/cf2 2023-10-27 23:53:14,154 INFO [StoreOpener-433bacad157e4d0e8f716e1942033cb2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 433bacad157e4d0e8f716e1942033cb2 columnFamilyName cf2 2023-10-27 23:53:14,155 INFO [StoreOpener-433bacad157e4d0e8f716e1942033cb2-1] regionserver.HStore(310): Store=433bacad157e4d0e8f716e1942033cb2/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:14,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:14,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:14,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:53:14,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:14,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 433bacad157e4d0e8f716e1942033cb2; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60603240, jitterRate=-0.09694135189056396}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:53:14,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 433bacad157e4d0e8f716e1942033cb2: 2023-10-27 23:53:14,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2., pid=61, masterSystemTime=1698450794130 2023-10-27 23:53:14,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2. 2023-10-27 23:53:14,163 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2. 2023-10-27 23:53:14,164 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=60 updating hbase:meta row=433bacad157e4d0e8f716e1942033cb2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:14,164 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450794164"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450794164"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450794164"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450794164"}]},"ts":"1698450794164"} 2023-10-27 23:53:14,166 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=61, resume processing ppid=60 2023-10-27 23:53:14,166 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=61, ppid=60, state=SUCCESS; OpenRegionProcedure 433bacad157e4d0e8f716e1942033cb2, server=jenkins-hbase16.apache.org,39059,1698450767147 in 187 msec 2023-10-27 23:53:14,167 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=60, resume processing ppid=56 2023-10-27 23:53:14,168 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=60, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=433bacad157e4d0e8f716e1942033cb2, ASSIGN in 343 msec 2023-10-27 23:53:14,168 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 in 549 msec 2023-10-27 23:53:14,374 DEBUG [hconnection-0x23f62efb-shared-pool-11] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-10-27 23:53:14,377 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58066, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-10-27 23:53:14,379 DEBUG [Listener at localhost/43373] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-10-27 23:53:14,383 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:53:14,384 INFO [Listener at localhost/43373] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionNoStoreFile 2023-10-27 23:53:14,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable testSplitTableRegionNoStoreFile 2023-10-27 23:53:14,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=63, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:53:14,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=63 2023-10-27 23:53:14,388 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450794388"}]},"ts":"1698450794388"} 2023-10-27 23:53:14,389 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=DISABLING in hbase:meta 2023-10-27 23:53:14,424 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionNoStoreFile to state=DISABLING 2023-10-27 23:53:14,425 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=c6d17e2577f36e5cbf594fca675240be, UNASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=433bacad157e4d0e8f716e1942033cb2, UNASSIGN}] 2023-10-27 23:53:14,426 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=433bacad157e4d0e8f716e1942033cb2, UNASSIGN 2023-10-27 23:53:14,426 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=433bacad157e4d0e8f716e1942033cb2, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:14,426 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450794426"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450794426"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450794426"}]},"ts":"1698450794426"} 2023-10-27 23:53:14,428 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=66, ppid=65, state=RUNNABLE; CloseRegionProcedure 433bacad157e4d0e8f716e1942033cb2, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:14,430 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=c6d17e2577f36e5cbf594fca675240be, UNASSIGN 2023-10-27 23:53:14,431 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=c6d17e2577f36e5cbf594fca675240be, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:14,431 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450794431"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450794431"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450794431"}]},"ts":"1698450794431"} 2023-10-27 23:53:14,432 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=67, ppid=64, state=RUNNABLE; CloseRegionProcedure c6d17e2577f36e5cbf594fca675240be, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:14,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=63 2023-10-27 23:53:14,579 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:14,580 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:14,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 433bacad157e4d0e8f716e1942033cb2, disabling compactions & flushes 2023-10-27 23:53:14,581 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2. 2023-10-27 23:53:14,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2. 2023-10-27 23:53:14,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2. after waiting 0 ms 2023-10-27 23:53:14,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2. 2023-10-27 23:53:14,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-10-27 23:53:14,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:53:14,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2. 2023-10-27 23:53:14,586 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 433bacad157e4d0e8f716e1942033cb2: 2023-10-27 23:53:14,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:14,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close c6d17e2577f36e5cbf594fca675240be 2023-10-27 23:53:14,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing c6d17e2577f36e5cbf594fca675240be, disabling compactions & flushes 2023-10-27 23:53:14,588 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be. 2023-10-27 23:53:14,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be. 2023-10-27 23:53:14,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be. after waiting 0 ms 2023-10-27 23:53:14,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be. 2023-10-27 23:53:14,588 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=433bacad157e4d0e8f716e1942033cb2, regionState=CLOSED 2023-10-27 23:53:14,589 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450794588"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450794588"}]},"ts":"1698450794588"} 2023-10-27 23:53:14,592 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=66, resume processing ppid=65 2023-10-27 23:53:14,592 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=66, ppid=65, state=SUCCESS; CloseRegionProcedure 433bacad157e4d0e8f716e1942033cb2, server=jenkins-hbase16.apache.org,39059,1698450767147 in 162 msec 2023-10-27 23:53:14,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-10-27 23:53:14,593 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=433bacad157e4d0e8f716e1942033cb2, UNASSIGN in 167 msec 2023-10-27 23:53:14,593 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:53:14,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be. 2023-10-27 23:53:14,594 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for c6d17e2577f36e5cbf594fca675240be: 2023-10-27 23:53:14,595 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed c6d17e2577f36e5cbf594fca675240be 2023-10-27 23:53:14,595 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=c6d17e2577f36e5cbf594fca675240be, regionState=CLOSED 2023-10-27 23:53:14,595 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450794595"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450794595"}]},"ts":"1698450794595"} 2023-10-27 23:53:14,597 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=67, resume processing ppid=64 2023-10-27 23:53:14,598 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=67, ppid=64, state=SUCCESS; CloseRegionProcedure c6d17e2577f36e5cbf594fca675240be, server=jenkins-hbase16.apache.org,39059,1698450767147 in 164 msec 2023-10-27 23:53:14,599 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=64, resume processing ppid=63 2023-10-27 23:53:14,599 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=c6d17e2577f36e5cbf594fca675240be, UNASSIGN in 173 msec 2023-10-27 23:53:14,600 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450794600"}]},"ts":"1698450794600"} 2023-10-27 23:53:14,601 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=DISABLED in hbase:meta 2023-10-27 23:53:14,611 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionNoStoreFile to state=DISABLED 2023-10-27 23:53:14,613 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile in 226 msec 2023-10-27 23:53:14,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=63 2023-10-27 23:53:14,691 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionNoStoreFile, procId: 63 completed 2023-10-27 23:53:14,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete testSplitTableRegionNoStoreFile 2023-10-27 23:53:14,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=68, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:53:14,694 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=68, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:53:14,695 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=68, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:53:14,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=68 2023-10-27 23:53:14,700 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:14,700 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be 2023-10-27 23:53:14,700 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:14,702 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2/recovered.edits] 2023-10-27 23:53:14,703 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be/recovered.edits] 2023-10-27 23:53:14,703 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15/recovered.edits] 2023-10-27 23:53:14,708 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2/recovered.edits/7.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2/recovered.edits/7.seqid 2023-10-27 23:53:14,708 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15/recovered.edits/4.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15/recovered.edits/4.seqid 2023-10-27 23:53:14,709 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be/recovered.edits/7.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be/recovered.edits/7.seqid 2023-10-27 23:53:14,709 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:53:14,709 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/aa542793a0b70c04a27ad5d5b58f9b15 2023-10-27 23:53:14,709 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionNoStoreFile/c6d17e2577f36e5cbf594fca675240be 2023-10-27 23:53:14,709 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionNoStoreFile regions 2023-10-27 23:53:14,711 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=68, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:53:14,713 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionNoStoreFile from hbase:meta 2023-10-27 23:53:14,715 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionNoStoreFile' descriptor. 2023-10-27 23:53:14,716 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=68, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:53:14,716 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionNoStoreFile' from region states. 2023-10-27 23:53:14,716 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450794716"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:14,716 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450794716"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:14,716 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450794716"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:14,718 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-10-27 23:53:14,718 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => aa542793a0b70c04a27ad5d5b58f9b15, NAME => 'testSplitTableRegionNoStoreFile,,1698450793117.aa542793a0b70c04a27ad5d5b58f9b15.', STARTKEY => '', ENDKEY => ''}, {ENCODED => c6d17e2577f36e5cbf594fca675240be, NAME => 'testSplitTableRegionNoStoreFile,,1698450793619.c6d17e2577f36e5cbf594fca675240be.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => 433bacad157e4d0e8f716e1942033cb2, NAME => 'testSplitTableRegionNoStoreFile,41,1698450793619.433bacad157e4d0e8f716e1942033cb2.', STARTKEY => '41', ENDKEY => ''}] 2023-10-27 23:53:14,718 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionNoStoreFile' as deleted. 2023-10-27 23:53:14,718 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1698450794718"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:14,720 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionNoStoreFile state from META 2023-10-27 23:53:14,728 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=68, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:53:14,729 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile in 36 msec 2023-10-27 23:53:14,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=68 2023-10-27 23:53:14,797 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionNoStoreFile, procId: 68 completed 2023-10-27 23:53:14,811 INFO [Listener at localhost/43373] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionNoStoreFile Thread=408 (was 401) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1185965544_17 at /127.0.0.1:37342 [Waiting for operation #26] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-metaLookup-shared--pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-metaLookup-shared--pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1185965544_17 at /127.0.0.1:43698 [Waiting for operation #24] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=867 (was 865) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=348 (was 348), ProcessCount=173 (was 171) - ProcessCount LEAK? -, AvailableMemoryMB=5524 (was 5525) 2023-10-27 23:53:14,826 INFO [Listener at localhost/43373] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionDeletedRowsDaughter Thread=408, OpenFileDescriptor=867, MaxFileDescriptor=60000, SystemLoadAverage=348, ProcessCount=173, AvailableMemoryMB=5523 2023-10-27 23:53:14,826 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:53:14,836 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-10-27 23:53:14,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(492): Client=jenkins//188.40.62.62 set balanceSwitch=false 2023-10-27 23:53:14,838 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=69, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:14,840 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-10-27 23:53:14,841 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-10-27 23:53:14,842 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:14,843 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2 empty. 2023-10-27 23:53:14,843 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:14,843 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionDeletedRowsDaughter regions 2023-10-27 23:53:14,862 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/.tabledesc/.tableinfo.0000000001 2023-10-27 23:53:14,864 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3087a119cbcd659122571a2b31e50ec2, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionDeletedRowsDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp 2023-10-27 23:53:14,879 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:14,879 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1604): Closing 3087a119cbcd659122571a2b31e50ec2, disabling compactions & flushes 2023-10-27 23:53:14,879 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:14,879 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:14,879 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. after waiting 0 ms 2023-10-27 23:53:14,879 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:14,879 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:14,879 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1558): Region close journal for 3087a119cbcd659122571a2b31e50ec2: 2023-10-27 23:53:14,882 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-10-27 23:53:14,882 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1698450794882"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450794882"}]},"ts":"1698450794882"} 2023-10-27 23:53:14,884 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-10-27 23:53:14,885 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-10-27 23:53:14,885 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450794885"}]},"ts":"1698450794885"} 2023-10-27 23:53:14,886 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=ENABLING in hbase:meta 2023-10-27 23:53:14,902 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-10-27 23:53:14,904 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-27 23:53:14,904 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-10-27 23:53:14,904 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-10-27 23:53:14,904 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-10-27 23:53:14,904 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=3087a119cbcd659122571a2b31e50ec2, ASSIGN}] 2023-10-27 23:53:14,905 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=3087a119cbcd659122571a2b31e50ec2, ASSIGN 2023-10-27 23:53:14,906 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=3087a119cbcd659122571a2b31e50ec2, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,39059,1698450767147; forceNewPlan=false, retain=false 2023-10-27 23:53:15,056 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-10-27 23:53:15,057 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=70 updating hbase:meta row=3087a119cbcd659122571a2b31e50ec2, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,057 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1698450795057"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450795057"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450795057"}]},"ts":"1698450795057"} 2023-10-27 23:53:15,059 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=71, ppid=70, state=RUNNABLE; OpenRegionProcedure 3087a119cbcd659122571a2b31e50ec2, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:15,211 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:15,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3087a119cbcd659122571a2b31e50ec2, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:53:15,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:53:15,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:15,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:15,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:15,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:15,216 INFO [StoreOpener-3087a119cbcd659122571a2b31e50ec2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:15,217 DEBUG [StoreOpener-3087a119cbcd659122571a2b31e50ec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1 2023-10-27 23:53:15,217 DEBUG [StoreOpener-3087a119cbcd659122571a2b31e50ec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1 2023-10-27 23:53:15,218 INFO [StoreOpener-3087a119cbcd659122571a2b31e50ec2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3087a119cbcd659122571a2b31e50ec2 columnFamilyName cf1 2023-10-27 23:53:15,219 INFO [StoreOpener-3087a119cbcd659122571a2b31e50ec2-1] regionserver.HStore(310): Store=3087a119cbcd659122571a2b31e50ec2/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:15,219 INFO [StoreOpener-3087a119cbcd659122571a2b31e50ec2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:15,220 DEBUG [StoreOpener-3087a119cbcd659122571a2b31e50ec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2 2023-10-27 23:53:15,220 DEBUG [StoreOpener-3087a119cbcd659122571a2b31e50ec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2 2023-10-27 23:53:15,221 INFO [StoreOpener-3087a119cbcd659122571a2b31e50ec2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3087a119cbcd659122571a2b31e50ec2 columnFamilyName cf2 2023-10-27 23:53:15,222 INFO [StoreOpener-3087a119cbcd659122571a2b31e50ec2-1] regionserver.HStore(310): Store=3087a119cbcd659122571a2b31e50ec2/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:15,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:15,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:15,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:53:15,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:15,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-10-27 23:53:15,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 3087a119cbcd659122571a2b31e50ec2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74193170, jitterRate=0.10556438565254211}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:53:15,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 3087a119cbcd659122571a2b31e50ec2: 2023-10-27 23:53:15,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2., pid=71, masterSystemTime=1698450795211 2023-10-27 23:53:15,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:15,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:15,233 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=70 updating hbase:meta row=3087a119cbcd659122571a2b31e50ec2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,233 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1698450795233"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450795233"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450795233"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450795233"}]},"ts":"1698450795233"} 2023-10-27 23:53:15,237 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=71, resume processing ppid=70 2023-10-27 23:53:15,237 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=71, ppid=70, state=SUCCESS; OpenRegionProcedure 3087a119cbcd659122571a2b31e50ec2, server=jenkins-hbase16.apache.org,39059,1698450767147 in 175 msec 2023-10-27 23:53:15,239 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=70, resume processing ppid=69 2023-10-27 23:53:15,239 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=70, ppid=69, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=3087a119cbcd659122571a2b31e50ec2, ASSIGN in 333 msec 2023-10-27 23:53:15,240 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-10-27 23:53:15,240 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450795240"}]},"ts":"1698450795240"} 2023-10-27 23:53:15,241 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=ENABLED in hbase:meta 2023-10-27 23:53:15,250 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-10-27 23:53:15,251 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 413 msec 2023-10-27 23:53:15,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:15,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,348 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:15,348 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:15,348 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-10-27 23:53:15,348 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-10-27 23:53:15,349 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,349 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,382 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,382 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,382 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,382 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,383 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,383 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:15,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:15,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:15,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:15,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,384 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:15,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,384 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:15,384 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:15,384 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:15,384 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:15,384 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:15,384 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:15,384 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:15,384 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:15,384 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:15,384 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:15,385 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:15,385 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:15,385 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:15,385 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:15,385 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:15,385 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. started... 2023-10-27 23:53:15,386 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3087a119cbcd659122571a2b31e50ec2 2/2 column families, dataSize=120 B heapSize=928 B 2023-10-27 23:53:15,394 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,402 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,402 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,402 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:15,403 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,403 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:15,403 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/63fc725845214be5b40378389ed66ac2 2023-10-27 23:53:15,424 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/7dfc15023e40482d9bf030d1dd1a1ca8 2023-10-27 23:53:15,430 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/63fc725845214be5b40378389ed66ac2 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/63fc725845214be5b40378389ed66ac2 2023-10-27 23:53:15,435 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/63fc725845214be5b40378389ed66ac2, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:53:15,436 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/7dfc15023e40482d9bf030d1dd1a1ca8 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7dfc15023e40482d9bf030d1dd1a1ca8 2023-10-27 23:53:15,440 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7dfc15023e40482d9bf030d1dd1a1ca8, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:53:15,440 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 3087a119cbcd659122571a2b31e50ec2 in 54ms, sequenceid=6, compaction requested=false 2023-10-27 23:53:15,441 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:15,441 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3087a119cbcd659122571a2b31e50ec2: 2023-10-27 23:53:15,441 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:15,441 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:15,441 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:15,441 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,441 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:15,441 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:15,452 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,452 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:15,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:15,453 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,453 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:15,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:15,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:15,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:15,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-10-27 23:53:15,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a7a438a[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:15,454 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-10-27 23:53:15,454 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,461 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,461 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,461 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,461 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,461 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:15,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,461 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:15,461 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:15,461 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,461 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:15,461 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:15,461 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:15,461 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:15,461 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:15,461 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:15,469 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:15,469 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:15,469 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:15,477 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:15,477 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,477 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:15,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:15,477 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:15,477 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:15,478 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:15,478 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:15,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:53:15,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:53:15,481 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-10-27 23:53:15,481 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:15,481 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:15,481 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,481 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:15,490 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,491 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,491 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,491 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,491 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,491 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:15,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,491 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:15,491 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,492 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,492 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,493 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:15,494 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,494 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,525 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,525 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,525 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,525 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:15,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:15,525 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,525 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,525 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,526 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,526 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,525 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,526 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,525 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,526 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:15,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,526 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,526 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,526 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,526 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,526 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,526 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,526 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,626 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:15,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:15,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:15,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:53:15,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,645 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:15,645 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:15,645 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-10-27 23:53:15,645 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-10-27 23:53:15,646 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,646 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,657 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,657 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,657 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,657 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,658 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,658 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:15,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,659 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:15,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:15,659 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:15,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:15,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,659 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:15,659 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:15,659 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:15,659 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:15,659 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:15,659 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:15,659 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:15,659 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:15,660 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:15,660 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:15,660 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:15,660 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:15,660 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:15,660 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. started... 2023-10-27 23:53:15,660 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:15,660 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3087a119cbcd659122571a2b31e50ec2 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:53:15,669 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,669 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/6a28b2e6aee54cff8d16b5f102b364d4 2023-10-27 23:53:15,677 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,677 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:15,677 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,678 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,678 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:15,682 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/3b48fbb9d13b42c3963d26aaac282a36 2023-10-27 23:53:15,686 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/6a28b2e6aee54cff8d16b5f102b364d4 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/6a28b2e6aee54cff8d16b5f102b364d4 2023-10-27 23:53:15,690 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/6a28b2e6aee54cff8d16b5f102b364d4, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:53:15,691 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/3b48fbb9d13b42c3963d26aaac282a36 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/3b48fbb9d13b42c3963d26aaac282a36 2023-10-27 23:53:15,695 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/3b48fbb9d13b42c3963d26aaac282a36, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:53:15,696 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 3087a119cbcd659122571a2b31e50ec2 in 36ms, sequenceid=19, compaction requested=false 2023-10-27 23:53:15,696 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3087a119cbcd659122571a2b31e50ec2: 2023-10-27 23:53:15,696 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:15,696 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:15,696 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:15,696 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,696 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:15,696 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:15,702 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,702 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:15,703 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,703 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:15,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-10-27 23:53:15,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dc8d497[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:15,705 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-10-27 23:53:15,705 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,711 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,711 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,711 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,711 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,711 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,711 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:15,711 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:15,711 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,711 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:15,711 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:15,711 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:15,711 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:15,711 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:15,711 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:15,719 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:15,719 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:15,719 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:15,719 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,719 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:15,719 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:15,719 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:15,727 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:15,727 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:15,727 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:15,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:53:15,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:53:15,731 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-10-27 23:53:15,731 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:15,731 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:15,731 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,731 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:15,741 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,741 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,741 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:15,741 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,741 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,741 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,741 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,741 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:15,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,742 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,742 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,744 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:15,745 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,745 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,752 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,752 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,752 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,752 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,752 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:15,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,752 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,752 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,753 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,752 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:15,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:15,753 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,753 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,753 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,753 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,753 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:15,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,753 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,753 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,753 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:15,753 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,754 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,754 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,854 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:15,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:15,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:15,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:53:15,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,869 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:15,869 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:15,869 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-10-27 23:53:15,869 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-10-27 23:53:15,870 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,870 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,877 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,877 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,877 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,877 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,878 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,878 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:15,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:15,879 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:15,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:15,879 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:15,879 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:15,879 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:15,879 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:15,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,879 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:15,879 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:15,879 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:15,879 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:15,879 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:15,879 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:15,879 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:15,879 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:15,880 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. started... 2023-10-27 23:53:15,879 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:15,880 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:15,880 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:15,880 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3087a119cbcd659122571a2b31e50ec2 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:53:15,888 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/cc065f1761f34463a1fbba299d0df2fc 2023-10-27 23:53:15,899 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/35754bb830684c3ea4bf4ae7eec26653 2023-10-27 23:53:15,903 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/cc065f1761f34463a1fbba299d0df2fc as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/cc065f1761f34463a1fbba299d0df2fc 2023-10-27 23:53:15,906 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,907 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/cc065f1761f34463a1fbba299d0df2fc, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:53:15,908 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/35754bb830684c3ea4bf4ae7eec26653 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/35754bb830684c3ea4bf4ae7eec26653 2023-10-27 23:53:15,912 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/35754bb830684c3ea4bf4ae7eec26653, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:53:15,913 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 3087a119cbcd659122571a2b31e50ec2 in 33ms, sequenceid=32, compaction requested=true 2023-10-27 23:53:15,913 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3087a119cbcd659122571a2b31e50ec2: 2023-10-27 23:53:15,913 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:15,913 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:15,913 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:15,913 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,913 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:15,913 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:15,919 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,919 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:15,919 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,927 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,927 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,927 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,927 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:15,928 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,928 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:15,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-10-27 23:53:15,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59cb32fa[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:15,930 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-10-27 23:53:15,930 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,941 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,941 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,941 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,941 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,941 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,941 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,941 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:15,941 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:15,941 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:15,941 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:15,941 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:15,941 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:15,942 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:15,942 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:15,949 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:15,949 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,949 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:15,949 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:15,957 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:15,957 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:15,957 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:15,957 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:15,957 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:15,958 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:53:15,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:53:15,960 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-10-27 23:53:15,960 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:15,960 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:15,961 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,961 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:15,969 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,969 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,969 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,969 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:15,969 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,969 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,969 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,969 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,970 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,970 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:15,972 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,972 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,972 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,982 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,982 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,982 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:15,982 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,982 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:15,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:15,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:15,982 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,982 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,982 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:15,982 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:15,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:15,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,983 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:15,983 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,983 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:15,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:15,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:15,983 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,984 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,984 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:15,984 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,984 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:15,984 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:15,984 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:15,984 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,083 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:16,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:16,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:16,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:53:16,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,099 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:16,099 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,099 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-10-27 23:53:16,099 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-10-27 23:53:16,100 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,100 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,107 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,107 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,107 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,107 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,108 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,108 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:16,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:16,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:16,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,109 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:16,109 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:16,109 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,109 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,109 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:16,109 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:16,109 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:16,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:16,109 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:16,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,109 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:16,109 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:16,109 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:16,109 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:16,109 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,110 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. started... 2023-10-27 23:53:16,110 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:16,110 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:16,110 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:16,110 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3087a119cbcd659122571a2b31e50ec2 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:53:16,118 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/777ad5ed8d484e1a8c4b6d196f849aee 2023-10-27 23:53:16,119 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,127 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,127 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,127 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:16,128 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,128 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:16,129 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/65cd201db5c84287825188b0aaaa3302 2023-10-27 23:53:16,133 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/777ad5ed8d484e1a8c4b6d196f849aee as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/777ad5ed8d484e1a8c4b6d196f849aee 2023-10-27 23:53:16,137 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/777ad5ed8d484e1a8c4b6d196f849aee, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:53:16,138 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/65cd201db5c84287825188b0aaaa3302 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/65cd201db5c84287825188b0aaaa3302 2023-10-27 23:53:16,141 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/65cd201db5c84287825188b0aaaa3302, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:53:16,142 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 3087a119cbcd659122571a2b31e50ec2 in 32ms, sequenceid=45, compaction requested=true 2023-10-27 23:53:16,142 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3087a119cbcd659122571a2b31e50ec2: 2023-10-27 23:53:16,142 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:16,142 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:16,142 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:16,142 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,143 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:16,143 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:16,152 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,152 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:16,153 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,153 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:16,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-10-27 23:53:16,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d796a15[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:16,155 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-10-27 23:53:16,155 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,166 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,166 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,166 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,166 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,166 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,166 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:16,166 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:16,166 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:16,166 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:16,166 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:16,166 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:16,167 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:16,167 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:16,166 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:16,194 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:16,194 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,194 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,194 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:16,194 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:16,194 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:16,194 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:16,194 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:16,194 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,195 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:53:16,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:53:16,197 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-10-27 23:53:16,197 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:16,197 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:16,197 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,197 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:16,211 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,211 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,211 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,211 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,211 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,211 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:16,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,211 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:16,211 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,212 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,212 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,213 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:16,214 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,214 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,224 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:16,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:16,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,225 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,225 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:16,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,225 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,225 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,225 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:16,225 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,225 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,225 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,225 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,325 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:16,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:16,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:16,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:53:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,343 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:16,343 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,343 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-10-27 23:53:16,343 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-10-27 23:53:16,343 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,343 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,352 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,352 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,352 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,352 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,353 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,353 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:16,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:16,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:16,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:16,354 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:16,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,354 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,354 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:16,354 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:16,354 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:16,354 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,354 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,354 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:16,355 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:16,354 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:16,355 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:16,354 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:16,355 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:16,355 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:16,355 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:16,355 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. started... 2023-10-27 23:53:16,355 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:16,355 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3087a119cbcd659122571a2b31e50ec2 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:53:16,361 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,367 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/afe14e8c7d85447c9755a312a463bc77 2023-10-27 23:53:16,369 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,369 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,369 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:16,369 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,369 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:16,383 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/40ac4a6fb99b46279e5e100d057e6817 2023-10-27 23:53:16,389 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/afe14e8c7d85447c9755a312a463bc77 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/afe14e8c7d85447c9755a312a463bc77 2023-10-27 23:53:16,393 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/afe14e8c7d85447c9755a312a463bc77, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:53:16,394 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/40ac4a6fb99b46279e5e100d057e6817 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/40ac4a6fb99b46279e5e100d057e6817 2023-10-27 23:53:16,397 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/40ac4a6fb99b46279e5e100d057e6817, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:53:16,398 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 3087a119cbcd659122571a2b31e50ec2 in 43ms, sequenceid=58, compaction requested=true 2023-10-27 23:53:16,398 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3087a119cbcd659122571a2b31e50ec2: 2023-10-27 23:53:16,398 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:16,398 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:16,398 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:16,398 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,398 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:16,398 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:16,407 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,407 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:16,408 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,408 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:16,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-10-27 23:53:16,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5370a23[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:16,409 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-10-27 23:53:16,409 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,419 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,419 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,419 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:16,420 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:16,420 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:16,420 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:16,420 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,420 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:16,420 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:16,420 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:16,421 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,421 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:16,421 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:16,421 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:16,427 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:16,427 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,427 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,428 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:16,436 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:16,436 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,436 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:16,444 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:16,444 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,444 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:53:16,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:53:16,447 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-10-27 23:53:16,447 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:16,447 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:16,447 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,447 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:16,457 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,457 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,457 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,457 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,458 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,457 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,458 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,458 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:16,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,458 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,459 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:16,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:16,460 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,460 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,460 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,487 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,487 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,487 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,487 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,487 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,487 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,487 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,487 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:16,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,487 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,487 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,487 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,488 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,488 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:16,488 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,488 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:16,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,488 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,488 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,488 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,488 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,488 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,588 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:16,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:16,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:16,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:53:16,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,605 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:16,605 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,605 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-10-27 23:53:16,605 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-10-27 23:53:16,606 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,606 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,616 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,616 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,616 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:16,617 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:16,617 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:16,618 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:16,618 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,617 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:16,618 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:16,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,618 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:16,618 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:16,618 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,618 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:16,618 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:16,618 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:16,618 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:16,618 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:16,618 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:16,618 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:16,618 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. started... 2023-10-27 23:53:16,618 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3087a119cbcd659122571a2b31e50ec2 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:53:16,627 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,629 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/1905c051309f490a8e66ea9f73348154 2023-10-27 23:53:16,635 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,636 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,636 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:16,636 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,636 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:16,645 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/eb872dc2a7004601b8b23acad657c0f4 2023-10-27 23:53:16,649 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/1905c051309f490a8e66ea9f73348154 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/1905c051309f490a8e66ea9f73348154 2023-10-27 23:53:16,653 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/1905c051309f490a8e66ea9f73348154, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:53:16,654 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/eb872dc2a7004601b8b23acad657c0f4 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/eb872dc2a7004601b8b23acad657c0f4 2023-10-27 23:53:16,658 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/eb872dc2a7004601b8b23acad657c0f4, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:53:16,659 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 3087a119cbcd659122571a2b31e50ec2 in 41ms, sequenceid=71, compaction requested=true 2023-10-27 23:53:16,659 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3087a119cbcd659122571a2b31e50ec2: 2023-10-27 23:53:16,659 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:16,659 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:16,659 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:16,659 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,659 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:16,659 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:16,669 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,669 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:16,670 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,670 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:16,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-10-27 23:53:16,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed01a7e[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:16,672 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-10-27 23:53:16,672 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,677 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,677 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,677 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,678 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,678 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:16,678 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:16,678 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:16,678 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,678 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:16,678 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:16,678 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:16,678 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:16,678 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:16,678 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:16,686 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,686 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:16,686 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,686 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:16,686 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:16,686 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:16,686 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,686 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:16,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:16,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:16,686 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,686 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:16,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:53:16,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:53:16,689 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-10-27 23:53:16,689 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:16,689 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:16,689 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,689 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:16,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:16,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,700 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:16,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,700 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,700 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:16,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,702 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,702 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:16,702 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,710 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,711 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,710 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,710 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,711 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,711 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,711 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,711 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:16,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:16,711 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,711 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,711 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,712 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:16,711 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,712 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:16,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,712 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,712 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,712 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,712 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,712 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,712 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,812 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:16,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:16,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:16,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:53:16,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,827 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:16,827 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,827 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-10-27 23:53:16,827 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-10-27 23:53:16,827 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,827 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,836 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,836 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,836 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,836 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,836 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,836 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:16,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:16,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:16,838 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:16,838 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. started... 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:16,838 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:16,838 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3087a119cbcd659122571a2b31e50ec2 2/2 column families, dataSize=530 B heapSize=2.33 KB 2023-10-27 23:53:16,844 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,852 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,852 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,852 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:16,853 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,853 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:16,854 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=265 B at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/0b0311fd9ed348efa26540e4a43f4ebf 2023-10-27 23:53:16,857 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0b0311fd9ed348efa26540e4a43f4ebf 2023-10-27 23:53:16,865 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=265 B at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816 2023-10-27 23:53:16,868 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cc4f9d94eccd4b2db7f8b7937dbc5816 2023-10-27 23:53:16,869 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/0b0311fd9ed348efa26540e4a43f4ebf as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/0b0311fd9ed348efa26540e4a43f4ebf 2023-10-27 23:53:16,874 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0b0311fd9ed348efa26540e4a43f4ebf 2023-10-27 23:53:16,874 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/0b0311fd9ed348efa26540e4a43f4ebf, entries=9, sequenceid=83, filesize=5.1 K 2023-10-27 23:53:16,874 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816 2023-10-27 23:53:16,879 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cc4f9d94eccd4b2db7f8b7937dbc5816 2023-10-27 23:53:16,879 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816, entries=9, sequenceid=83, filesize=5.1 K 2023-10-27 23:53:16,880 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~530 B/530, heapSize ~2.30 KB/2352, currentSize=0 B/0 for 3087a119cbcd659122571a2b31e50ec2 in 42ms, sequenceid=83, compaction requested=true 2023-10-27 23:53:16,880 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3087a119cbcd659122571a2b31e50ec2: 2023-10-27 23:53:16,880 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:16,880 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:16,880 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:16,880 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,880 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:16,880 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:16,891 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,891 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:16,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:16,891 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,891 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:16,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:16,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-10-27 23:53:16,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e0ca5bc[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:16,893 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-10-27 23:53:16,893 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,902 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,902 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,902 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,903 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,903 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,903 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:16,903 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:16,903 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,903 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:16,903 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:16,903 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:16,903 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:16,903 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:16,903 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:16,930 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:16,930 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,930 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:16,941 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:16,941 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,941 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:16,941 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:16,941 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,941 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,941 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:53:16,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:53:16,944 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-10-27 23:53:16,944 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:16,944 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:16,944 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,944 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:16,952 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,952 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,952 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,952 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:16,952 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,952 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,952 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,953 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:16,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,953 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,954 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,955 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:16,955 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,955 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,966 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,966 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,966 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,966 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:16,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:16,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:16,966 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:16,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,966 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,966 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,966 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:16,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:16,966 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:16,966 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:16,966 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,967 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:16,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:16,967 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:16,967 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,967 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:16,967 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,967 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:16,967 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:16,967 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:16,967 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,067 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:17,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:17,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:17,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:53:17,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,077 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:17,078 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:17,078 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-10-27 23:53:17,078 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-10-27 23:53:17,078 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,078 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,086 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,086 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:17,086 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:17,086 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:17,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:17,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:17,086 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,086 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:17,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:17,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:17,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:17,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:17,088 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:17,088 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:17,088 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. started... 2023-10-27 23:53:17,088 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3087a119cbcd659122571a2b31e50ec2 2/2 column families, dataSize=250 B heapSize=1.52 KB 2023-10-27 23:53:17,094 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,096 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/91bba55d5dd141b9b834844bbe04221d 2023-10-27 23:53:17,100 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 91bba55d5dd141b9b834844bbe04221d 2023-10-27 23:53:17,102 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,102 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,102 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:17,102 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,103 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:17,108 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/d134cf340ef84f9e812dd37cf5af4726 2023-10-27 23:53:17,112 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d134cf340ef84f9e812dd37cf5af4726 2023-10-27 23:53:17,113 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/91bba55d5dd141b9b834844bbe04221d as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/91bba55d5dd141b9b834844bbe04221d 2023-10-27 23:53:17,117 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 91bba55d5dd141b9b834844bbe04221d 2023-10-27 23:53:17,117 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/91bba55d5dd141b9b834844bbe04221d, entries=5, sequenceid=91, filesize=5.0 K 2023-10-27 23:53:17,118 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/d134cf340ef84f9e812dd37cf5af4726 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/d134cf340ef84f9e812dd37cf5af4726 2023-10-27 23:53:17,123 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d134cf340ef84f9e812dd37cf5af4726 2023-10-27 23:53:17,124 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/d134cf340ef84f9e812dd37cf5af4726, entries=5, sequenceid=91, filesize=5.0 K 2023-10-27 23:53:17,124 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~250 B/250, heapSize ~1.48 KB/1520, currentSize=0 B/0 for 3087a119cbcd659122571a2b31e50ec2 in 36ms, sequenceid=91, compaction requested=true 2023-10-27 23:53:17,125 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3087a119cbcd659122571a2b31e50ec2: 2023-10-27 23:53:17,125 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:17,125 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:17,125 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:17,125 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,125 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:17,125 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:17,136 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,136 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:17,136 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,136 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:17,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-10-27 23:53:17,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68435faa[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:17,138 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-10-27 23:53:17,138 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,144 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,144 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,144 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,144 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:17,144 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:17,144 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:17,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,144 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:17,144 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:17,144 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,145 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:17,144 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,145 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:17,145 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:17,145 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:17,152 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:17,152 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:17,152 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:17,161 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:17,161 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:17,161 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:17,161 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:17,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:17,169 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:17,169 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:17,169 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:17,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:53:17,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:53:17,173 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-10-27 23:53:17,173 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:17,173 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:17,173 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,173 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:17,203 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,203 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:17,203 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,204 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:17,203 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,204 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:17,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,204 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,204 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:17,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:17,205 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,205 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,206 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:17,206 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,207 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,244 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,244 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:17,244 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:17,244 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:17,244 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:17,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:17,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:17,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:17,244 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:17,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:17,244 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:17,244 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,245 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:17,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:17,245 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:17,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:17,245 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:17,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:17,245 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:17,245 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:17,245 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,245 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,245 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,246 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,246 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,246 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,246 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,345 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:17,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:17,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:53:17,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:53:17,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,356 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:17,356 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:17,356 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-10-27 23:53:17,356 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-10-27 23:53:17,356 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,356 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,369 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:17,369 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:17,369 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,369 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:17,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:17,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:17,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:17,369 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,370 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:17,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:17,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:17,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:17,371 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:17,371 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:17,371 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. started... 2023-10-27 23:53:17,371 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3087a119cbcd659122571a2b31e50ec2 2/2 column families, dataSize=250 B heapSize=1.52 KB 2023-10-27 23:53:17,377 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,377 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,378 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,378 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:17,378 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,378 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:17,379 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/9fe07c4c375140d9834d2ff75c430845 2023-10-27 23:53:17,383 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9fe07c4c375140d9834d2ff75c430845 2023-10-27 23:53:17,391 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/431c0549e98f4986b0a97348165aabf9 2023-10-27 23:53:17,395 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 431c0549e98f4986b0a97348165aabf9 2023-10-27 23:53:17,396 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/9fe07c4c375140d9834d2ff75c430845 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/9fe07c4c375140d9834d2ff75c430845 2023-10-27 23:53:17,400 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9fe07c4c375140d9834d2ff75c430845 2023-10-27 23:53:17,400 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/9fe07c4c375140d9834d2ff75c430845, entries=5, sequenceid=99, filesize=5.0 K 2023-10-27 23:53:17,401 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/431c0549e98f4986b0a97348165aabf9 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/431c0549e98f4986b0a97348165aabf9 2023-10-27 23:53:17,405 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 431c0549e98f4986b0a97348165aabf9 2023-10-27 23:53:17,405 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/431c0549e98f4986b0a97348165aabf9, entries=5, sequenceid=99, filesize=5.0 K 2023-10-27 23:53:17,406 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~250 B/250, heapSize ~1.48 KB/1520, currentSize=0 B/0 for 3087a119cbcd659122571a2b31e50ec2 in 35ms, sequenceid=99, compaction requested=true 2023-10-27 23:53:17,406 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3087a119cbcd659122571a2b31e50ec2: 2023-10-27 23:53:17,406 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:17,406 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:53:17,406 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:53:17,406 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,406 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-10-27 23:53:17,406 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-10-27 23:53:17,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,416 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:17,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:53:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:17,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-10-27 23:53:17,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55183693[Count = 0] remaining members to acquire global barrier 2023-10-27 23:53:17,418 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-10-27 23:53:17,418 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,427 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,427 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,427 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,428 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,428 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:53:17,428 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:17,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,428 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:17,428 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:53:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,428 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:17,428 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-10-27 23:53:17,428 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:17,428 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:53:17,428 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-10-27 23:53:17,428 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:53:17,436 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:17,436 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,436 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:17,436 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:17,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:17,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:17,444 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:17,444 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:17,444 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:17,452 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-10-27 23:53:17,452 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:17,453 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-10-27 23:53:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:53:17,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:53:17,455 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-10-27 23:53:17,455 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:53:17,455 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:53:17,455 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,455 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:53:17,466 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,466 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,466 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,466 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,466 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:53:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:53:17,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,466 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:17,466 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:17,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,466 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:53:17,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:17,467 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,467 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:53:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,468 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:53:17,468 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,468 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,477 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:17,477 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:17,477 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:53:17,477 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:53:17,477 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:17,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-10-27 23:53:17,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:53:17,478 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:17,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:17,477 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:17,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:53:17,477 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:53:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:53:17,478 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:53:17,478 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,478 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:53:17,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:17,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:17,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:53:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:17,478 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:17,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:53:17,478 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,479 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,479 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,479 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,479 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:53:17,479 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:17,479 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,479 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:17,578 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:53:17,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:53:17,582 DEBUG [Listener at localhost/43373] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,582 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:17,583 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:17,583 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:17,583 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:17,585 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=73, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=3087a119cbcd659122571a2b31e50ec2, UNASSIGN}] 2023-10-27 23:53:17,586 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=73, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=3087a119cbcd659122571a2b31e50ec2, UNASSIGN 2023-10-27 23:53:17,586 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=73 updating hbase:meta row=3087a119cbcd659122571a2b31e50ec2, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,586 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1698450797586"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450797586"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450797586"}]},"ts":"1698450797586"} 2023-10-27 23:53:17,587 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=74, ppid=73, state=RUNNABLE; CloseRegionProcedure 3087a119cbcd659122571a2b31e50ec2, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:17,739 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:17,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 3087a119cbcd659122571a2b31e50ec2, disabling compactions & flushes 2023-10-27 23:53:17,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:17,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:17,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. after waiting 0 ms 2023-10-27 23:53:17,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:17,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 3087a119cbcd659122571a2b31e50ec2 2/2 column families, dataSize=50 B heapSize=720 B 2023-10-27 23:53:17,748 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=25 B at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/61d1feafbe854977a083041e6e811db2 2023-10-27 23:53:17,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 61d1feafbe854977a083041e6e811db2 2023-10-27 23:53:17,763 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=25 B at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/7811752907234ed6bee370bd314c903e 2023-10-27 23:53:17,766 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7811752907234ed6bee370bd314c903e 2023-10-27 23:53:17,767 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf1/61d1feafbe854977a083041e6e811db2 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/61d1feafbe854977a083041e6e811db2 2023-10-27 23:53:17,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 61d1feafbe854977a083041e6e811db2 2023-10-27 23:53:17,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/61d1feafbe854977a083041e6e811db2, entries=1, sequenceid=103, filesize=4.9 K 2023-10-27 23:53:17,772 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.tmp/cf2/7811752907234ed6bee370bd314c903e as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7811752907234ed6bee370bd314c903e 2023-10-27 23:53:17,776 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7811752907234ed6bee370bd314c903e 2023-10-27 23:53:17,776 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7811752907234ed6bee370bd314c903e, entries=1, sequenceid=103, filesize=4.9 K 2023-10-27 23:53:17,776 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~50 B/50, heapSize ~688 B/688, currentSize=0 B/0 for 3087a119cbcd659122571a2b31e50ec2 in 36ms, sequenceid=103, compaction requested=true 2023-10-27 23:53:17,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=1 2023-10-27 23:53:17,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:53:17,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2. 2023-10-27 23:53:17,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 3087a119cbcd659122571a2b31e50ec2: 2023-10-27 23:53:17,783 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,784 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=73 updating hbase:meta row=3087a119cbcd659122571a2b31e50ec2, regionState=CLOSED 2023-10-27 23:53:17,784 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1698450797784"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450797784"}]},"ts":"1698450797784"} 2023-10-27 23:53:17,786 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=74, resume processing ppid=73 2023-10-27 23:53:17,787 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=74, ppid=73, state=SUCCESS; CloseRegionProcedure 3087a119cbcd659122571a2b31e50ec2, server=jenkins-hbase16.apache.org,39059,1698450767147 in 198 msec 2023-10-27 23:53:17,788 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=73, resume processing ppid=72 2023-10-27 23:53:17,788 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=73, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=3087a119cbcd659122571a2b31e50ec2, UNASSIGN in 202 msec 2023-10-27 23:53:17,800 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=72 splitting 20 storefiles, region=3087a119cbcd659122571a2b31e50ec2, threads=16 2023-10-27 23:53:17,801 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/35754bb830684c3ea4bf4ae7eec26653 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,801 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/3b48fbb9d13b42c3963d26aaac282a36 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,801 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/40ac4a6fb99b46279e5e100d057e6817 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,801 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/431c0549e98f4986b0a97348165aabf9 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,801 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/65cd201db5c84287825188b0aaaa3302 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,801 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7811752907234ed6bee370bd314c903e for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,801 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7dfc15023e40482d9bf030d1dd1a1ca8 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,801 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,801 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/d134cf340ef84f9e812dd37cf5af4726 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,802 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/eb872dc2a7004601b8b23acad657c0f4 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,802 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/0b0311fd9ed348efa26540e4a43f4ebf for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,802 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/1905c051309f490a8e66ea9f73348154 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,802 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/63fc725845214be5b40378389ed66ac2 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,803 DEBUG [StoreFileSplitter-pool-14] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/6a28b2e6aee54cff8d16b5f102b364d4 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,804 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/61d1feafbe854977a083041e6e811db2 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,807 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/777ad5ed8d484e1a8c4b6d196f849aee for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,811 INFO [StoreFileSplitter-pool-7] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cc4f9d94eccd4b2db7f8b7937dbc5816 2023-10-27 23:53:17,811 INFO [StoreFileSplitter-pool-3] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 431c0549e98f4986b0a97348165aabf9 2023-10-27 23:53:17,815 INFO [StoreFileSplitter-pool-5] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7811752907234ed6bee370bd314c903e 2023-10-27 23:53:17,815 DEBUG [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/1905c051309f490a8e66ea9f73348154, top=false 2023-10-27 23:53:17,815 DEBUG [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/65cd201db5c84287825188b0aaaa3302, top=false 2023-10-27 23:53:17,816 DEBUG [StoreFileSplitter-pool-15] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/777ad5ed8d484e1a8c4b6d196f849aee, top=false 2023-10-27 23:53:17,817 INFO [StoreFileSplitter-pool-8] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d134cf340ef84f9e812dd37cf5af4726 2023-10-27 23:53:17,819 INFO [StoreFileSplitter-pool-3] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 431c0549e98f4986b0a97348165aabf9 2023-10-27 23:53:17,819 DEBUG [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/431c0549e98f4986b0a97348165aabf9, top=true 2023-10-27 23:53:17,819 DEBUG [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/eb872dc2a7004601b8b23acad657c0f4, top=false 2023-10-27 23:53:17,822 INFO [StoreFileSplitter-pool-12] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 61d1feafbe854977a083041e6e811db2 2023-10-27 23:53:17,824 INFO [StoreFileSplitter-pool-5] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7811752907234ed6bee370bd314c903e 2023-10-27 23:53:17,824 DEBUG [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7811752907234ed6bee370bd314c903e, top=true 2023-10-27 23:53:17,826 INFO [StoreFileSplitter-pool-12] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 61d1feafbe854977a083041e6e811db2 2023-10-27 23:53:17,826 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/61d1feafbe854977a083041e6e811db2, top=true 2023-10-27 23:53:17,843 INFO [StoreFileSplitter-pool-10] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0b0311fd9ed348efa26540e4a43f4ebf 2023-10-27 23:53:17,844 DEBUG [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/40ac4a6fb99b46279e5e100d057e6817, top=false 2023-10-27 23:53:17,845 INFO [StoreFileSplitter-pool-8] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d134cf340ef84f9e812dd37cf5af4726 2023-10-27 23:53:17,845 DEBUG [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/d134cf340ef84f9e812dd37cf5af4726, top=true 2023-10-27 23:53:17,857 INFO [StoreFileSplitter-pool-15] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.splits/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-777ad5ed8d484e1a8c4b6d196f849aee for child: f0da177e9a77436613e559466a9befb3, parent: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,857 INFO [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.splits/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-1905c051309f490a8e66ea9f73348154 for child: f0da177e9a77436613e559466a9befb3, parent: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,860 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/777ad5ed8d484e1a8c4b6d196f849aee for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,861 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/1905c051309f490a8e66ea9f73348154 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,861 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/91bba55d5dd141b9b834844bbe04221d for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,861 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/9fe07c4c375140d9834d2ff75c430845 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,864 INFO [StoreFileSplitter-pool-11] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9fe07c4c375140d9834d2ff75c430845 2023-10-27 23:53:17,864 INFO [StoreFileSplitter-pool-15] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 91bba55d5dd141b9b834844bbe04221d 2023-10-27 23:53:17,872 INFO [StoreFileSplitter-pool-15] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 91bba55d5dd141b9b834844bbe04221d 2023-10-27 23:53:17,872 DEBUG [StoreFileSplitter-pool-15] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/91bba55d5dd141b9b834844bbe04221d, top=true 2023-10-27 23:53:17,873 INFO [StoreFileSplitter-pool-11] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9fe07c4c375140d9834d2ff75c430845 2023-10-27 23:53:17,873 DEBUG [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/9fe07c4c375140d9834d2ff75c430845, top=true 2023-10-27 23:53:17,891 INFO [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.splits/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-65cd201db5c84287825188b0aaaa3302 for child: f0da177e9a77436613e559466a9befb3, parent: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,895 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/65cd201db5c84287825188b0aaaa3302 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,895 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/afe14e8c7d85447c9755a312a463bc77 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,898 DEBUG [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/afe14e8c7d85447c9755a312a463bc77, top=false 2023-10-27 23:53:17,904 INFO [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.splits/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-d134cf340ef84f9e812dd37cf5af4726 for child: f3419eb3ecb19fbd75b9ef498fab9b83, parent: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,905 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/d134cf340ef84f9e812dd37cf5af4726 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,905 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/cc065f1761f34463a1fbba299d0df2fc for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,911 INFO [StoreFileSplitter-pool-7] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cc4f9d94eccd4b2db7f8b7937dbc5816 2023-10-27 23:53:17,920 INFO [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.splits/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-40ac4a6fb99b46279e5e100d057e6817 for child: f0da177e9a77436613e559466a9befb3, parent: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,920 INFO [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.splits/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-eb872dc2a7004601b8b23acad657c0f4 for child: f0da177e9a77436613e559466a9befb3, parent: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,924 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.splits/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-61d1feafbe854977a083041e6e811db2 for child: f3419eb3ecb19fbd75b9ef498fab9b83, parent: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,924 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/61d1feafbe854977a083041e6e811db2 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,924 INFO [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.splits/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-7811752907234ed6bee370bd314c903e for child: f3419eb3ecb19fbd75b9ef498fab9b83, parent: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,924 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7811752907234ed6bee370bd314c903e for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,925 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/eb872dc2a7004601b8b23acad657c0f4 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,935 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/40ac4a6fb99b46279e5e100d057e6817 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,946 INFO [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.splits/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-431c0549e98f4986b0a97348165aabf9 for child: f3419eb3ecb19fbd75b9ef498fab9b83, parent: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,946 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/431c0549e98f4986b0a97348165aabf9 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,953 INFO [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.splits/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-9fe07c4c375140d9834d2ff75c430845 for child: f3419eb3ecb19fbd75b9ef498fab9b83, parent: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,953 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/9fe07c4c375140d9834d2ff75c430845 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,959 INFO [StoreFileSplitter-pool-15] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.splits/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-91bba55d5dd141b9b834844bbe04221d for child: f3419eb3ecb19fbd75b9ef498fab9b83, parent: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,959 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/91bba55d5dd141b9b834844bbe04221d for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,972 INFO [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/.splits/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-afe14e8c7d85447c9755a312a463bc77 for child: f0da177e9a77436613e559466a9befb3, parent: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,976 INFO [StoreFileSplitter-pool-10] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0b0311fd9ed348efa26540e4a43f4ebf 2023-10-27 23:53:17,977 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/afe14e8c7d85447c9755a312a463bc77 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:17,993 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:18,007 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7dfc15023e40482d9bf030d1dd1a1ca8 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:18,011 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/35754bb830684c3ea4bf4ae7eec26653 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:18,019 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/0b0311fd9ed348efa26540e4a43f4ebf for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:18,032 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/63fc725845214be5b40378389ed66ac2 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:18,035 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/cc065f1761f34463a1fbba299d0df2fc for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:18,037 DEBUG [StoreFileSplitter-pool-14] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/6a28b2e6aee54cff8d16b5f102b364d4 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:18,405 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/3b48fbb9d13b42c3963d26aaac282a36 for region: 3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:18,405 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=72 split storefiles for region 3087a119cbcd659122571a2b31e50ec2 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-10-27 23:53:18,447 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=-1 2023-10-27 23:53:18,449 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=-1 2023-10-27 23:53:18,451 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1698450798451"},{"qualifier":"splitA","vlen":75,"tag":[],"timestamp":"1698450798451"},{"qualifier":"splitB","vlen":75,"tag":[],"timestamp":"1698450798451"}]},"ts":"1698450798451"} 2023-10-27 23:53:18,451 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1698450798451"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450798451"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450798451"}]},"ts":"1698450798451"} 2023-10-27 23:53:18,451 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1698450798451"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450798451"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450798451"}]},"ts":"1698450798451"} 2023-10-27 23:53:18,455 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f0da177e9a77436613e559466a9befb3, ASSIGN}, {pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f3419eb3ecb19fbd75b9ef498fab9b83, ASSIGN}] 2023-10-27 23:53:18,456 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f3419eb3ecb19fbd75b9ef498fab9b83, ASSIGN 2023-10-27 23:53:18,463 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f3419eb3ecb19fbd75b9ef498fab9b83, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase16.apache.org,39059,1698450767147; forceNewPlan=false, retain=false 2023-10-27 23:53:18,464 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f0da177e9a77436613e559466a9befb3, ASSIGN 2023-10-27 23:53:18,465 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f0da177e9a77436613e559466a9befb3, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase16.apache.org,39059,1698450767147; forceNewPlan=false, retain=false 2023-10-27 23:53:18,613 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-10-27 23:53:18,618 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=75 updating hbase:meta row=f0da177e9a77436613e559466a9befb3, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:18,618 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1698450798618"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450798618"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450798618"}]},"ts":"1698450798618"} 2023-10-27 23:53:18,622 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=77, ppid=75, state=RUNNABLE; OpenRegionProcedure f0da177e9a77436613e559466a9befb3, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:18,635 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=76 updating hbase:meta row=f3419eb3ecb19fbd75b9ef498fab9b83, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:18,635 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1698450798635"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450798635"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450798635"}]},"ts":"1698450798635"} 2023-10-27 23:53:18,636 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=78, ppid=76, state=RUNNABLE; OpenRegionProcedure f3419eb3ecb19fbd75b9ef498fab9b83, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:18,776 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:18,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:18,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f0da177e9a77436613e559466a9befb3, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.', STARTKEY => '', ENDKEY => '60'} 2023-10-27 23:53:18,780 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:53:18,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter f0da177e9a77436613e559466a9befb3 2023-10-27 23:53:18,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:18,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for f0da177e9a77436613e559466a9befb3 2023-10-27 23:53:18,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for f0da177e9a77436613e559466a9befb3 2023-10-27 23:53:18,781 INFO [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region f0da177e9a77436613e559466a9befb3 2023-10-27 23:53:18,782 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1 2023-10-27 23:53:18,782 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1 2023-10-27 23:53:18,782 INFO [StoreOpener-f0da177e9a77436613e559466a9befb3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f0da177e9a77436613e559466a9befb3 columnFamilyName cf1 2023-10-27 23:53:18,791 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:18,792 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/0b0311fd9ed348efa26540e4a43f4ebf-bottom 2023-10-27 23:53:18,797 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/63fc725845214be5b40378389ed66ac2.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/63fc725845214be5b40378389ed66ac2-bottom 2023-10-27 23:53:18,801 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/6a28b2e6aee54cff8d16b5f102b364d4.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/6a28b2e6aee54cff8d16b5f102b364d4-bottom 2023-10-27 23:53:18,805 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/cc065f1761f34463a1fbba299d0df2fc.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/cc065f1761f34463a1fbba299d0df2fc-bottom 2023-10-27 23:53:18,810 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-1905c051309f490a8e66ea9f73348154 2023-10-27 23:53:18,813 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-777ad5ed8d484e1a8c4b6d196f849aee 2023-10-27 23:53:18,816 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-afe14e8c7d85447c9755a312a463bc77 2023-10-27 23:53:18,816 INFO [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(310): Store=f0da177e9a77436613e559466a9befb3/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:18,817 INFO [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region f0da177e9a77436613e559466a9befb3 2023-10-27 23:53:18,817 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2 2023-10-27 23:53:18,818 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2 2023-10-27 23:53:18,818 INFO [StoreOpener-f0da177e9a77436613e559466a9befb3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f0da177e9a77436613e559466a9befb3 columnFamilyName cf2 2023-10-27 23:53:18,830 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/35754bb830684c3ea4bf4ae7eec26653.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/35754bb830684c3ea4bf4ae7eec26653-bottom 2023-10-27 23:53:18,834 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/3b48fbb9d13b42c3963d26aaac282a36.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/3b48fbb9d13b42c3963d26aaac282a36-bottom 2023-10-27 23:53:18,839 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/7dfc15023e40482d9bf030d1dd1a1ca8.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7dfc15023e40482d9bf030d1dd1a1ca8-bottom 2023-10-27 23:53:18,842 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:18,844 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816-bottom 2023-10-27 23:53:18,848 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-40ac4a6fb99b46279e5e100d057e6817 2023-10-27 23:53:18,852 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-65cd201db5c84287825188b0aaaa3302 2023-10-27 23:53:18,856 DEBUG [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-eb872dc2a7004601b8b23acad657c0f4 2023-10-27 23:53:18,856 INFO [StoreOpener-f0da177e9a77436613e559466a9befb3-1] regionserver.HStore(310): Store=f0da177e9a77436613e559466a9befb3/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:18,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3 2023-10-27 23:53:18,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3 2023-10-27 23:53:18,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:53:18,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for f0da177e9a77436613e559466a9befb3 2023-10-27 23:53:18,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened f0da177e9a77436613e559466a9befb3; next sequenceid=107; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59129548, jitterRate=-0.1189010739326477}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:53:18,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for f0da177e9a77436613e559466a9befb3: 2023-10-27 23:53:18,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3., pid=77, masterSystemTime=1698450798776 2023-10-27 23:53:18,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:18,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:53:18,864 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:53:18,864 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:53:18,864 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:18,864 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:18,864 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1912): f0da177e9a77436613e559466a9befb3/cf1 is initiating minor compaction (all files) 2023-10-27 23:53:18,864 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1912): f0da177e9a77436613e559466a9befb3/cf2 is initiating minor compaction (all files) 2023-10-27 23:53:18,864 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f0da177e9a77436613e559466a9befb3/cf1 in testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:18,864 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f0da177e9a77436613e559466a9befb3/cf2 in testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:18,864 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/63fc725845214be5b40378389ed66ac2.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/63fc725845214be5b40378389ed66ac2-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/6a28b2e6aee54cff8d16b5f102b364d4.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/6a28b2e6aee54cff8d16b5f102b364d4-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/cc065f1761f34463a1fbba299d0df2fc.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/cc065f1761f34463a1fbba299d0df2fc-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-777ad5ed8d484e1a8c4b6d196f849aee, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-afe14e8c7d85447c9755a312a463bc77, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-1905c051309f490a8e66ea9f73348154, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/0b0311fd9ed348efa26540e4a43f4ebf-bottom] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/.tmp, totalSize=35.0 K 2023-10-27 23:53:18,864 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/7dfc15023e40482d9bf030d1dd1a1ca8.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7dfc15023e40482d9bf030d1dd1a1ca8-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/3b48fbb9d13b42c3963d26aaac282a36.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/3b48fbb9d13b42c3963d26aaac282a36-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/35754bb830684c3ea4bf4ae7eec26653.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/35754bb830684c3ea4bf4ae7eec26653-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-65cd201db5c84287825188b0aaaa3302, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-40ac4a6fb99b46279e5e100d057e6817, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-eb872dc2a7004601b8b23acad657c0f4, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816-bottom] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/.tmp, totalSize=35.0 K 2023-10-27 23:53:18,865 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting 63fc725845214be5b40378389ed66ac2.3087a119cbcd659122571a2b31e50ec2, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450795341 2023-10-27 23:53:18,865 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting 7dfc15023e40482d9bf030d1dd1a1ca8.3087a119cbcd659122571a2b31e50ec2, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450795341 2023-10-27 23:53:18,865 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting 6a28b2e6aee54cff8d16b5f102b364d4.3087a119cbcd659122571a2b31e50ec2, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450795629 2023-10-27 23:53:18,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:18,866 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:18,866 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:18,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f3419eb3ecb19fbd75b9ef498fab9b83, NAME => 'testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.', STARTKEY => '60', ENDKEY => ''} 2023-10-27 23:53:18,866 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:53:18,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:18,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:18,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:18,867 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting 3b48fbb9d13b42c3963d26aaac282a36.3087a119cbcd659122571a2b31e50ec2, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450795629 2023-10-27 23:53:18,867 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting cc065f1761f34463a1fbba299d0df2fc.3087a119cbcd659122571a2b31e50ec2, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450795855 2023-10-27 23:53:18,867 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=75 updating hbase:meta row=f0da177e9a77436613e559466a9befb3, regionState=OPEN, openSeqNum=107, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:18,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:18,867 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1698450798867"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450798867"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450798867"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450798867"}]},"ts":"1698450798867"} 2023-10-27 23:53:18,867 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting 35754bb830684c3ea4bf4ae7eec26653.3087a119cbcd659122571a2b31e50ec2, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450795855 2023-10-27 23:53:18,868 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-65cd201db5c84287825188b0aaaa3302, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450796085 2023-10-27 23:53:18,868 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-40ac4a6fb99b46279e5e100d057e6817, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450796327 2023-10-27 23:53:18,869 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-eb872dc2a7004601b8b23acad657c0f4, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450796590 2023-10-27 23:53:18,869 INFO [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:18,869 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1698450796814 2023-10-27 23:53:18,871 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=77, resume processing ppid=75 2023-10-27 23:53:18,871 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=77, ppid=75, state=SUCCESS; OpenRegionProcedure f0da177e9a77436613e559466a9befb3, server=jenkins-hbase16.apache.org,39059,1698450767147 in 247 msec 2023-10-27 23:53:18,872 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1 2023-10-27 23:53:18,872 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=75, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f0da177e9a77436613e559466a9befb3, ASSIGN in 416 msec 2023-10-27 23:53:18,872 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-777ad5ed8d484e1a8c4b6d196f849aee, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450796085 2023-10-27 23:53:18,872 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-afe14e8c7d85447c9755a312a463bc77, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450796327 2023-10-27 23:53:18,872 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1 2023-10-27 23:53:18,873 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-1905c051309f490a8e66ea9f73348154, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450796590 2023-10-27 23:53:18,873 INFO [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f3419eb3ecb19fbd75b9ef498fab9b83 columnFamilyName cf1 2023-10-27 23:53:18,874 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting 0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1698450796814 2023-10-27 23:53:18,884 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:18,884 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/0b0311fd9ed348efa26540e4a43f4ebf-top 2023-10-27 23:53:18,890 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/63fc725845214be5b40378389ed66ac2.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/63fc725845214be5b40378389ed66ac2-top 2023-10-27 23:53:18,895 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/6a28b2e6aee54cff8d16b5f102b364d4.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/6a28b2e6aee54cff8d16b5f102b364d4-top 2023-10-27 23:53:18,895 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] throttle.PressureAwareThroughputController(145): f0da177e9a77436613e559466a9befb3#cf2#compaction#72 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:18,899 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] throttle.PressureAwareThroughputController(145): f0da177e9a77436613e559466a9befb3#cf1#compaction#73 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:18,904 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/cc065f1761f34463a1fbba299d0df2fc.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/cc065f1761f34463a1fbba299d0df2fc-top 2023-10-27 23:53:18,908 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 61d1feafbe854977a083041e6e811db2 2023-10-27 23:53:18,908 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-61d1feafbe854977a083041e6e811db2 2023-10-27 23:53:18,913 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 91bba55d5dd141b9b834844bbe04221d 2023-10-27 23:53:18,913 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-91bba55d5dd141b9b834844bbe04221d 2023-10-27 23:53:18,916 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9fe07c4c375140d9834d2ff75c430845 2023-10-27 23:53:18,917 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-9fe07c4c375140d9834d2ff75c430845 2023-10-27 23:53:18,917 INFO [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(310): Store=f3419eb3ecb19fbd75b9ef498fab9b83/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:18,917 INFO [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:18,917 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/.tmp/cf2/175d138ef8f04ee6af1c598e8d9ee3df as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/175d138ef8f04ee6af1c598e8d9ee3df 2023-10-27 23:53:18,917 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2 2023-10-27 23:53:18,917 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2 2023-10-27 23:53:18,918 INFO [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f3419eb3ecb19fbd75b9ef498fab9b83 columnFamilyName cf2 2023-10-27 23:53:18,921 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/.tmp/cf1/27e36191615f4a53a0d709facb56a11e as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/27e36191615f4a53a0d709facb56a11e 2023-10-27 23:53:18,921 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in f0da177e9a77436613e559466a9befb3/cf2 of f0da177e9a77436613e559466a9befb3 into 175d138ef8f04ee6af1c598e8d9ee3df(size=6.8 K), total size for store is 6.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:18,921 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f0da177e9a77436613e559466a9befb3: 2023-10-27 23:53:18,921 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3., storeName=f0da177e9a77436613e559466a9befb3/cf2, priority=9, startTime=1698450798863; duration=0sec 2023-10-27 23:53:18,921 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:18,926 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in f0da177e9a77436613e559466a9befb3/cf1 of f0da177e9a77436613e559466a9befb3 into 27e36191615f4a53a0d709facb56a11e(size=6.8 K), total size for store is 6.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:18,926 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f0da177e9a77436613e559466a9befb3: 2023-10-27 23:53:18,926 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3., storeName=f0da177e9a77436613e559466a9befb3/cf1, priority=9, startTime=1698450798863; duration=0sec 2023-10-27 23:53:18,926 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:18,927 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/35754bb830684c3ea4bf4ae7eec26653.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/35754bb830684c3ea4bf4ae7eec26653-top 2023-10-27 23:53:18,931 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/3b48fbb9d13b42c3963d26aaac282a36.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/3b48fbb9d13b42c3963d26aaac282a36-top 2023-10-27 23:53:18,940 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/7dfc15023e40482d9bf030d1dd1a1ca8.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7dfc15023e40482d9bf030d1dd1a1ca8-top 2023-10-27 23:53:18,944 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:18,945 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816-top 2023-10-27 23:53:18,949 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 431c0549e98f4986b0a97348165aabf9 2023-10-27 23:53:18,949 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-431c0549e98f4986b0a97348165aabf9 2023-10-27 23:53:18,953 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7811752907234ed6bee370bd314c903e 2023-10-27 23:53:18,953 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-7811752907234ed6bee370bd314c903e 2023-10-27 23:53:18,956 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d134cf340ef84f9e812dd37cf5af4726 2023-10-27 23:53:18,956 DEBUG [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-d134cf340ef84f9e812dd37cf5af4726 2023-10-27 23:53:18,957 INFO [StoreOpener-f3419eb3ecb19fbd75b9ef498fab9b83-1] regionserver.HStore(310): Store=f3419eb3ecb19fbd75b9ef498fab9b83/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:18,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:18,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:18,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:53:18,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:18,962 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened f3419eb3ecb19fbd75b9ef498fab9b83; next sequenceid=107; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67401319, jitterRate=0.004357919096946716}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:53:18,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for f3419eb3ecb19fbd75b9ef498fab9b83: 2023-10-27 23:53:18,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83., pid=78, masterSystemTime=1698450798776 2023-10-27 23:53:18,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:18,964 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:53:18,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:53:18,964 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:53:18,964 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:18,964 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:18,964 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1912): f3419eb3ecb19fbd75b9ef498fab9b83/cf1 is initiating minor compaction (all files) 2023-10-27 23:53:18,964 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1912): f3419eb3ecb19fbd75b9ef498fab9b83/cf2 is initiating minor compaction (all files) 2023-10-27 23:53:18,964 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f3419eb3ecb19fbd75b9ef498fab9b83/cf1 in testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:18,964 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f3419eb3ecb19fbd75b9ef498fab9b83/cf2 in testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:18,964 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/7dfc15023e40482d9bf030d1dd1a1ca8.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7dfc15023e40482d9bf030d1dd1a1ca8-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/3b48fbb9d13b42c3963d26aaac282a36.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/3b48fbb9d13b42c3963d26aaac282a36-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/35754bb830684c3ea4bf4ae7eec26653.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/35754bb830684c3ea4bf4ae7eec26653-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-d134cf340ef84f9e812dd37cf5af4726, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-431c0549e98f4986b0a97348165aabf9, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-7811752907234ed6bee370bd314c903e] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/.tmp, totalSize=34.8 K 2023-10-27 23:53:18,964 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/63fc725845214be5b40378389ed66ac2.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/63fc725845214be5b40378389ed66ac2-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/6a28b2e6aee54cff8d16b5f102b364d4.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/6a28b2e6aee54cff8d16b5f102b364d4-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/cc065f1761f34463a1fbba299d0df2fc.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/cc065f1761f34463a1fbba299d0df2fc-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/0b0311fd9ed348efa26540e4a43f4ebf-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-91bba55d5dd141b9b834844bbe04221d, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-9fe07c4c375140d9834d2ff75c430845, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-61d1feafbe854977a083041e6e811db2] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/.tmp, totalSize=34.8 K 2023-10-27 23:53:18,965 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting 7dfc15023e40482d9bf030d1dd1a1ca8.3087a119cbcd659122571a2b31e50ec2, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1698450795341 2023-10-27 23:53:18,965 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting 63fc725845214be5b40378389ed66ac2.3087a119cbcd659122571a2b31e50ec2, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1698450795341 2023-10-27 23:53:18,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:18,966 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:18,966 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting 3b48fbb9d13b42c3963d26aaac282a36.3087a119cbcd659122571a2b31e50ec2, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1698450795629 2023-10-27 23:53:18,966 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting 6a28b2e6aee54cff8d16b5f102b364d4.3087a119cbcd659122571a2b31e50ec2, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1698450795629 2023-10-27 23:53:18,966 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=76 updating hbase:meta row=f3419eb3ecb19fbd75b9ef498fab9b83, regionState=OPEN, openSeqNum=107, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:18,966 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting 35754bb830684c3ea4bf4ae7eec26653.3087a119cbcd659122571a2b31e50ec2, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1698450795855 2023-10-27 23:53:18,966 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting cc065f1761f34463a1fbba299d0df2fc.3087a119cbcd659122571a2b31e50ec2, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1698450795855 2023-10-27 23:53:18,967 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1698450798966"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450798966"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450798966"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450798966"}]},"ts":"1698450798966"} 2023-10-27 23:53:18,967 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1698450796814 2023-10-27 23:53:18,967 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting 0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1698450796814 2023-10-27 23:53:18,967 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-d134cf340ef84f9e812dd37cf5af4726, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=9223372036854775807 2023-10-27 23:53:18,967 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-431c0549e98f4986b0a97348165aabf9, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=9223372036854775807 2023-10-27 23:53:18,967 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-91bba55d5dd141b9b834844bbe04221d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=9223372036854775807 2023-10-27 23:53:18,968 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-7811752907234ed6bee370bd314c903e, keycount=1, bloomtype=ROW, size=4.9 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-10-27 23:53:18,968 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-9fe07c4c375140d9834d2ff75c430845, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=9223372036854775807 2023-10-27 23:53:18,968 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-61d1feafbe854977a083041e6e811db2, keycount=1, bloomtype=ROW, size=4.9 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-10-27 23:53:18,970 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=78, resume processing ppid=76 2023-10-27 23:53:18,970 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=78, ppid=76, state=SUCCESS; OpenRegionProcedure f3419eb3ecb19fbd75b9ef498fab9b83, server=jenkins-hbase16.apache.org,39059,1698450767147 in 332 msec 2023-10-27 23:53:18,972 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=76, resume processing ppid=72 2023-10-27 23:53:18,972 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=76, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f3419eb3ecb19fbd75b9ef498fab9b83, ASSIGN in 515 msec 2023-10-27 23:53:18,974 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 in 1.3900 sec 2023-10-27 23:53:18,982 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] throttle.PressureAwareThroughputController(145): f3419eb3ecb19fbd75b9ef498fab9b83#cf2#compaction#74 average throughput is 0.00 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:18,982 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] throttle.PressureAwareThroughputController(145): f3419eb3ecb19fbd75b9ef498fab9b83#cf1#compaction#75 average throughput is 0.00 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:18,986 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-27 23:53:18,994 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 7af136e520044fec816090bd23338fcb: NONE, but ROW specified in column family configuration 2023-10-27 23:53:18,995 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/.tmp/cf1/7af136e520044fec816090bd23338fcb as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/7af136e520044fec816090bd23338fcb 2023-10-27 23:53:19,003 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 7af136e520044fec816090bd23338fcb: NONE, but ROW specified in column family configuration 2023-10-27 23:53:19,003 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 75feb78fed924ca3be5dda3f54b9f818: NONE, but ROW specified in column family configuration 2023-10-27 23:53:19,004 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/.tmp/cf2/75feb78fed924ca3be5dda3f54b9f818 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/75feb78fed924ca3be5dda3f54b9f818 2023-10-27 23:53:19,004 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in f3419eb3ecb19fbd75b9ef498fab9b83/cf1 of f3419eb3ecb19fbd75b9ef498fab9b83 into 7af136e520044fec816090bd23338fcb(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:19,004 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f3419eb3ecb19fbd75b9ef498fab9b83: 2023-10-27 23:53:19,004 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83., storeName=f3419eb3ecb19fbd75b9ef498fab9b83/cf1, priority=9, startTime=1698450798963; duration=0sec 2023-10-27 23:53:19,004 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:19,007 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 75feb78fed924ca3be5dda3f54b9f818: NONE, but ROW specified in column family configuration 2023-10-27 23:53:19,008 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in f3419eb3ecb19fbd75b9ef498fab9b83/cf2 of f3419eb3ecb19fbd75b9ef498fab9b83 into 75feb78fed924ca3be5dda3f54b9f818(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:19,008 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f3419eb3ecb19fbd75b9ef498fab9b83: 2023-10-27 23:53:19,008 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83., storeName=f3419eb3ecb19fbd75b9ef498fab9b83/cf2, priority=9, startTime=1698450798964; duration=0sec 2023-10-27 23:53:19,009 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:19,095 DEBUG [Listener at localhost/43373] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-10-27 23:53:19,101 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:58078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-10-27 23:53:19,101 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39059] regionserver.RSRpcServices(1662): Compacting testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:19,102 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39059] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-10-27 23:53:19,102 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39059] regionserver.HStore(1912): f0da177e9a77436613e559466a9befb3/cf1 is initiating major compaction (all files) 2023-10-27 23:53:19,103 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39059] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@2fd40a3c; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:19,103 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f0da177e9a77436613e559466a9befb3/cf1 in testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:19,103 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39059] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-10-27 23:53:19,103 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/27e36191615f4a53a0d709facb56a11e] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/.tmp, totalSize=6.8 K 2023-10-27 23:53:19,103 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39059] regionserver.HStore(1912): f0da177e9a77436613e559466a9befb3/cf2 is initiating major compaction (all files) 2023-10-27 23:53:19,103 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39059] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@63565f1f; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:53:19,103 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f0da177e9a77436613e559466a9befb3/cf2 in testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:19,103 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/175d138ef8f04ee6af1c598e8d9ee3df] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/.tmp, totalSize=6.8 K 2023-10-27 23:53:19,103 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting 27e36191615f4a53a0d709facb56a11e, keycount=49, bloomtype=ROW, size=6.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1698450795341 2023-10-27 23:53:19,104 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting 175d138ef8f04ee6af1c598e8d9ee3df, keycount=49, bloomtype=ROW, size=6.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1698450795341 2023-10-27 23:53:19,106 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.RSRpcServices(1662): Compacting testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:19,106 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-10-27 23:53:19,106 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.HStore(1912): f3419eb3ecb19fbd75b9ef498fab9b83/cf1 is initiating major compaction (all files) 2023-10-27 23:53:19,106 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@33e2b3f0; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:53:19,106 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-10-27 23:53:19,106 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.HStore(1912): f3419eb3ecb19fbd75b9ef498fab9b83/cf2 is initiating major compaction (all files) 2023-10-27 23:53:19,106 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@4ce17ad3; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-10-27 23:53:19,107 INFO [Listener at localhost/43373] hbase.Waiter(180): Waiting up to [6,000] milli-secs(wait.for.ratio=[1]) 2023-10-27 23:53:19,109 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] throttle.PressureAwareThroughputController(145): f0da177e9a77436613e559466a9befb3#cf2#compaction#76 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:19,109 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] throttle.PressureAwareThroughputController(145): f0da177e9a77436613e559466a9befb3#cf1#compaction#77 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:19,130 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39059] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. is not mergeable because it has references 2023-10-27 23:53:19,131 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/.tmp/cf2/68fb04efa1f7406e8b18e2a0041c6832 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/68fb04efa1f7406e8b18e2a0041c6832 2023-10-27 23:53:19,131 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. is not mergeable because it has references 2023-10-27 23:53:19,131 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/.tmp/cf1/6069d3f139424c2782ed5fb275d363ea as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/6069d3f139424c2782ed5fb275d363ea 2023-10-27 23:53:19,136 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in f0da177e9a77436613e559466a9befb3/cf1 of f0da177e9a77436613e559466a9befb3 into 6069d3f139424c2782ed5fb275d363ea(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:19,136 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in f0da177e9a77436613e559466a9befb3/cf2 of f0da177e9a77436613e559466a9befb3 into 68fb04efa1f7406e8b18e2a0041c6832(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:19,136 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f0da177e9a77436613e559466a9befb3: 2023-10-27 23:53:19,136 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f0da177e9a77436613e559466a9befb3: 2023-10-27 23:53:19,137 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3., storeName=cf1, fileCount=1, fileSize=6.8 K (6.8 K), priority=1, time=1698450799102; duration=0sec 2023-10-27 23:53:19,137 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3., storeName=cf2, fileCount=1, fileSize=6.8 K (6.8 K), priority=1, time=1698450799103; duration=0sec 2023-10-27 23:53:19,138 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-10-27 23:53:19,138 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-10-27 23:53:19,138 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f3419eb3ecb19fbd75b9ef498fab9b83/cf1 in testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:19,138 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f3419eb3ecb19fbd75b9ef498fab9b83/cf2 in testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:19,138 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/7af136e520044fec816090bd23338fcb] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/.tmp, totalSize=5.0 K 2023-10-27 23:53:19,138 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/75feb78fed924ca3be5dda3f54b9f818] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/.tmp, totalSize=5.0 K 2023-10-27 23:53:19,138 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting 7af136e520044fec816090bd23338fcb, keycount=0, bloomtype=NONE, size=5.0 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-10-27 23:53:19,138 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting 75feb78fed924ca3be5dda3f54b9f818, keycount=0, bloomtype=NONE, size=5.0 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-10-27 23:53:19,141 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] throttle.PressureAwareThroughputController(145): f3419eb3ecb19fbd75b9ef498fab9b83#cf2#compaction#79 average throughput is NaN MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:19,141 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] throttle.PressureAwareThroughputController(145): f3419eb3ecb19fbd75b9ef498fab9b83#cf1#compaction#78 average throughput is NaN MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:53:19,157 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 5d6d4f77eec54f3ebff6b6ecc0f10ac6: NONE, but ROW specified in column family configuration 2023-10-27 23:53:19,157 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for a449f58b146e4c04bffa2bfb86172374: NONE, but ROW specified in column family configuration 2023-10-27 23:53:19,158 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/.tmp/cf1/a449f58b146e4c04bffa2bfb86172374 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/a449f58b146e4c04bffa2bfb86172374 2023-10-27 23:53:19,158 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/.tmp/cf2/5d6d4f77eec54f3ebff6b6ecc0f10ac6 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/5d6d4f77eec54f3ebff6b6ecc0f10ac6 2023-10-27 23:53:19,162 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for a449f58b146e4c04bffa2bfb86172374: NONE, but ROW specified in column family configuration 2023-10-27 23:53:19,163 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 5d6d4f77eec54f3ebff6b6ecc0f10ac6: NONE, but ROW specified in column family configuration 2023-10-27 23:53:19,164 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in f3419eb3ecb19fbd75b9ef498fab9b83/cf2 of f3419eb3ecb19fbd75b9ef498fab9b83 into 5d6d4f77eec54f3ebff6b6ecc0f10ac6(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:19,164 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in f3419eb3ecb19fbd75b9ef498fab9b83/cf1 of f3419eb3ecb19fbd75b9ef498fab9b83 into a449f58b146e4c04bffa2bfb86172374(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:53:19,164 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f3419eb3ecb19fbd75b9ef498fab9b83: 2023-10-27 23:53:19,164 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f3419eb3ecb19fbd75b9ef498fab9b83: 2023-10-27 23:53:19,164 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83., storeName=cf2, fileCount=1, fileSize=5.0 K (5.0 K), priority=1, time=1698450799106; duration=0sec 2023-10-27 23:53:19,164 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83., storeName=cf1, fileCount=1, fileSize=5.0 K (5.0 K), priority=1, time=1698450799106; duration=0sec 2023-10-27 23:53:19,164 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:19,164 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:53:19,243 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39059] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. is not mergeable because it has references 2023-10-27 23:53:19,244 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. is not mergeable because it has references 2023-10-27 23:53:19,251 DEBUG [Listener at localhost/43373] client.ClientScanner(240): Advancing internal scanner to startKey at '60', inclusive 2023-10-27 23:53:19,253 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:53:19,255 INFO [Listener at localhost/43373] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:19,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:19,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=79, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:19,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=79 2023-10-27 23:53:19,260 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450799260"}]},"ts":"1698450799260"} 2023-10-27 23:53:19,261 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=DISABLING in hbase:meta 2023-10-27 23:53:19,302 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionDeletedRowsDaughter to state=DISABLING 2023-10-27 23:53:19,303 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=80, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f0da177e9a77436613e559466a9befb3, UNASSIGN}, {pid=81, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f3419eb3ecb19fbd75b9ef498fab9b83, UNASSIGN}] 2023-10-27 23:53:19,304 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f3419eb3ecb19fbd75b9ef498fab9b83, UNASSIGN 2023-10-27 23:53:19,305 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=f3419eb3ecb19fbd75b9ef498fab9b83, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:19,305 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1698450799305"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450799305"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450799305"}]},"ts":"1698450799305"} 2023-10-27 23:53:19,306 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=82, ppid=81, state=RUNNABLE; CloseRegionProcedure f3419eb3ecb19fbd75b9ef498fab9b83, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:19,308 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f0da177e9a77436613e559466a9befb3, UNASSIGN 2023-10-27 23:53:19,309 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=f0da177e9a77436613e559466a9befb3, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:19,309 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1698450799309"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450799309"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450799309"}]},"ts":"1698450799309"} 2023-10-27 23:53:19,310 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=83, ppid=80, state=RUNNABLE; CloseRegionProcedure f0da177e9a77436613e559466a9befb3, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:19,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=79 2023-10-27 23:53:19,458 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:19,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:19,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing f3419eb3ecb19fbd75b9ef498fab9b83, disabling compactions & flushes 2023-10-27 23:53:19,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:19,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:19,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. after waiting 0 ms 2023-10-27 23:53:19,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:19,459 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/63fc725845214be5b40378389ed66ac2.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/63fc725845214be5b40378389ed66ac2-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/6a28b2e6aee54cff8d16b5f102b364d4.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/6a28b2e6aee54cff8d16b5f102b364d4-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/cc065f1761f34463a1fbba299d0df2fc.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/cc065f1761f34463a1fbba299d0df2fc-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/0b0311fd9ed348efa26540e4a43f4ebf-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-91bba55d5dd141b9b834844bbe04221d, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-9fe07c4c375140d9834d2ff75c430845, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/7af136e520044fec816090bd23338fcb, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-61d1feafbe854977a083041e6e811db2] to archive 2023-10-27 23:53:19,460 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:53:19,462 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/63fc725845214be5b40378389ed66ac2.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/63fc725845214be5b40378389ed66ac2.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,463 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/6a28b2e6aee54cff8d16b5f102b364d4.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/6a28b2e6aee54cff8d16b5f102b364d4.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,464 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/cc065f1761f34463a1fbba299d0df2fc.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/cc065f1761f34463a1fbba299d0df2fc.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,466 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,467 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-91bba55d5dd141b9b834844bbe04221d to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-91bba55d5dd141b9b834844bbe04221d 2023-10-27 23:53:19,468 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-9fe07c4c375140d9834d2ff75c430845 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-9fe07c4c375140d9834d2ff75c430845 2023-10-27 23:53:19,469 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/7af136e520044fec816090bd23338fcb to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/7af136e520044fec816090bd23338fcb 2023-10-27 23:53:19,470 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-61d1feafbe854977a083041e6e811db2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-61d1feafbe854977a083041e6e811db2 2023-10-27 23:53:19,471 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/7dfc15023e40482d9bf030d1dd1a1ca8.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7dfc15023e40482d9bf030d1dd1a1ca8-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/3b48fbb9d13b42c3963d26aaac282a36.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/3b48fbb9d13b42c3963d26aaac282a36-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/35754bb830684c3ea4bf4ae7eec26653.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/35754bb830684c3ea4bf4ae7eec26653-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-d134cf340ef84f9e812dd37cf5af4726, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-431c0549e98f4986b0a97348165aabf9, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/75feb78fed924ca3be5dda3f54b9f818, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-7811752907234ed6bee370bd314c903e] to archive 2023-10-27 23:53:19,472 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:53:19,473 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/7dfc15023e40482d9bf030d1dd1a1ca8.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/7dfc15023e40482d9bf030d1dd1a1ca8.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,474 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/3b48fbb9d13b42c3963d26aaac282a36.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/3b48fbb9d13b42c3963d26aaac282a36.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,475 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/35754bb830684c3ea4bf4ae7eec26653.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/35754bb830684c3ea4bf4ae7eec26653.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,476 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,477 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-d134cf340ef84f9e812dd37cf5af4726 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-d134cf340ef84f9e812dd37cf5af4726 2023-10-27 23:53:19,478 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-431c0549e98f4986b0a97348165aabf9 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-431c0549e98f4986b0a97348165aabf9 2023-10-27 23:53:19,479 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/75feb78fed924ca3be5dda3f54b9f818 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/75feb78fed924ca3be5dda3f54b9f818 2023-10-27 23:53:19,479 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-7811752907234ed6bee370bd314c903e to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-7811752907234ed6bee370bd314c903e 2023-10-27 23:53:19,483 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/recovered.edits/113.seqid, newMaxSeqId=113, maxSeqId=106 2023-10-27 23:53:19,483 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:53:19,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83. 2023-10-27 23:53:19,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for f3419eb3ecb19fbd75b9ef498fab9b83: 2023-10-27 23:53:19,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:19,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close f0da177e9a77436613e559466a9befb3 2023-10-27 23:53:19,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing f0da177e9a77436613e559466a9befb3, disabling compactions & flushes 2023-10-27 23:53:19,486 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=f3419eb3ecb19fbd75b9ef498fab9b83, regionState=CLOSED 2023-10-27 23:53:19,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:19,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:19,486 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1698450799486"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450799486"}]},"ts":"1698450799486"} 2023-10-27 23:53:19,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. after waiting 0 ms 2023-10-27 23:53:19,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:19,488 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/63fc725845214be5b40378389ed66ac2.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/63fc725845214be5b40378389ed66ac2-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/6a28b2e6aee54cff8d16b5f102b364d4.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/6a28b2e6aee54cff8d16b5f102b364d4-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/cc065f1761f34463a1fbba299d0df2fc.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/cc065f1761f34463a1fbba299d0df2fc-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-777ad5ed8d484e1a8c4b6d196f849aee, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-afe14e8c7d85447c9755a312a463bc77, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-1905c051309f490a8e66ea9f73348154, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/27e36191615f4a53a0d709facb56a11e, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/0b0311fd9ed348efa26540e4a43f4ebf-bottom] to archive 2023-10-27 23:53:19,489 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:53:19,490 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=82, resume processing ppid=81 2023-10-27 23:53:19,490 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=82, ppid=81, state=SUCCESS; CloseRegionProcedure f3419eb3ecb19fbd75b9ef498fab9b83, server=jenkins-hbase16.apache.org,39059,1698450767147 in 182 msec 2023-10-27 23:53:19,491 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/63fc725845214be5b40378389ed66ac2.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/63fc725845214be5b40378389ed66ac2.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,491 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=81, ppid=79, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f3419eb3ecb19fbd75b9ef498fab9b83, UNASSIGN in 187 msec 2023-10-27 23:53:19,493 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/6a28b2e6aee54cff8d16b5f102b364d4.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/6a28b2e6aee54cff8d16b5f102b364d4.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,493 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/cc065f1761f34463a1fbba299d0df2fc.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/cc065f1761f34463a1fbba299d0df2fc.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,494 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-777ad5ed8d484e1a8c4b6d196f849aee to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-777ad5ed8d484e1a8c4b6d196f849aee 2023-10-27 23:53:19,495 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-afe14e8c7d85447c9755a312a463bc77 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-afe14e8c7d85447c9755a312a463bc77 2023-10-27 23:53:19,496 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-1905c051309f490a8e66ea9f73348154 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-1905c051309f490a8e66ea9f73348154 2023-10-27 23:53:19,497 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/27e36191615f4a53a0d709facb56a11e to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/27e36191615f4a53a0d709facb56a11e 2023-10-27 23:53:19,498 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/0b0311fd9ed348efa26540e4a43f4ebf.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,500 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/7dfc15023e40482d9bf030d1dd1a1ca8.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7dfc15023e40482d9bf030d1dd1a1ca8-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/3b48fbb9d13b42c3963d26aaac282a36.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/3b48fbb9d13b42c3963d26aaac282a36-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/35754bb830684c3ea4bf4ae7eec26653.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/35754bb830684c3ea4bf4ae7eec26653-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-65cd201db5c84287825188b0aaaa3302, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-40ac4a6fb99b46279e5e100d057e6817, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-eb872dc2a7004601b8b23acad657c0f4, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/175d138ef8f04ee6af1c598e8d9ee3df, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816-bottom] to archive 2023-10-27 23:53:19,500 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:53:19,501 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/7dfc15023e40482d9bf030d1dd1a1ca8.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/7dfc15023e40482d9bf030d1dd1a1ca8.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,502 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/3b48fbb9d13b42c3963d26aaac282a36.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/3b48fbb9d13b42c3963d26aaac282a36.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,503 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/35754bb830684c3ea4bf4ae7eec26653.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/35754bb830684c3ea4bf4ae7eec26653.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,504 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-65cd201db5c84287825188b0aaaa3302 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-65cd201db5c84287825188b0aaaa3302 2023-10-27 23:53:19,505 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-40ac4a6fb99b46279e5e100d057e6817 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-40ac4a6fb99b46279e5e100d057e6817 2023-10-27 23:53:19,505 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-eb872dc2a7004601b8b23acad657c0f4 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/testSplitTableRegionDeletedRowsDaughter=3087a119cbcd659122571a2b31e50ec2-eb872dc2a7004601b8b23acad657c0f4 2023-10-27 23:53:19,506 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/175d138ef8f04ee6af1c598e8d9ee3df to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/175d138ef8f04ee6af1c598e8d9ee3df 2023-10-27 23:53:19,507 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816.3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/recovered.edits/113.seqid, newMaxSeqId=113, maxSeqId=106 2023-10-27 23:53:19,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:53:19,512 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3. 2023-10-27 23:53:19,512 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for f0da177e9a77436613e559466a9befb3: 2023-10-27 23:53:19,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed f0da177e9a77436613e559466a9befb3 2023-10-27 23:53:19,513 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=f0da177e9a77436613e559466a9befb3, regionState=CLOSED 2023-10-27 23:53:19,514 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1698450799513"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450799513"}]},"ts":"1698450799513"} 2023-10-27 23:53:19,516 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=83, resume processing ppid=80 2023-10-27 23:53:19,516 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=83, ppid=80, state=SUCCESS; CloseRegionProcedure f0da177e9a77436613e559466a9befb3, server=jenkins-hbase16.apache.org,39059,1698450767147 in 205 msec 2023-10-27 23:53:19,518 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=80, resume processing ppid=79 2023-10-27 23:53:19,518 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=80, ppid=79, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f0da177e9a77436613e559466a9befb3, UNASSIGN in 213 msec 2023-10-27 23:53:19,518 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450799518"}]},"ts":"1698450799518"} 2023-10-27 23:53:19,519 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=DISABLED in hbase:meta 2023-10-27 23:53:19,527 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionDeletedRowsDaughter to state=DISABLED 2023-10-27 23:53:19,529 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 272 msec 2023-10-27 23:53:19,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=79 2023-10-27 23:53:19,562 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionDeletedRowsDaughter, procId: 79 completed 2023-10-27 23:53:19,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:19,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=84, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:19,566 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=84, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:19,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=84 2023-10-27 23:53:19,566 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=84, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:19,571 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,571 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:19,571 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3 2023-10-27 23:53:19,573 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/recovered.edits] 2023-10-27 23:53:19,573 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/recovered.edits] 2023-10-27 23:53:19,573 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/recovered.edits] 2023-10-27 23:53:19,575 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/6069d3f139424c2782ed5fb275d363ea to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf1/6069d3f139424c2782ed5fb275d363ea 2023-10-27 23:53:19,575 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/a449f58b146e4c04bffa2bfb86172374 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf1/a449f58b146e4c04bffa2bfb86172374 2023-10-27 23:53:19,575 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/0b0311fd9ed348efa26540e4a43f4ebf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/0b0311fd9ed348efa26540e4a43f4ebf 2023-10-27 23:53:19,576 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/1905c051309f490a8e66ea9f73348154 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/1905c051309f490a8e66ea9f73348154 2023-10-27 23:53:19,577 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/68fb04efa1f7406e8b18e2a0041c6832 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/cf2/68fb04efa1f7406e8b18e2a0041c6832 2023-10-27 23:53:19,577 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/5d6d4f77eec54f3ebff6b6ecc0f10ac6 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/cf2/5d6d4f77eec54f3ebff6b6ecc0f10ac6 2023-10-27 23:53:19,578 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/61d1feafbe854977a083041e6e811db2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/61d1feafbe854977a083041e6e811db2 2023-10-27 23:53:19,579 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/63fc725845214be5b40378389ed66ac2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/63fc725845214be5b40378389ed66ac2 2023-10-27 23:53:19,580 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/6a28b2e6aee54cff8d16b5f102b364d4 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/6a28b2e6aee54cff8d16b5f102b364d4 2023-10-27 23:53:19,580 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/recovered.edits/113.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3/recovered.edits/113.seqid 2023-10-27 23:53:19,580 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/recovered.edits/113.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83/recovered.edits/113.seqid 2023-10-27 23:53:19,580 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f0da177e9a77436613e559466a9befb3 2023-10-27 23:53:19,580 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:53:19,581 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/777ad5ed8d484e1a8c4b6d196f849aee to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/777ad5ed8d484e1a8c4b6d196f849aee 2023-10-27 23:53:19,582 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/91bba55d5dd141b9b834844bbe04221d to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/91bba55d5dd141b9b834844bbe04221d 2023-10-27 23:53:19,583 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/9fe07c4c375140d9834d2ff75c430845 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/9fe07c4c375140d9834d2ff75c430845 2023-10-27 23:53:19,584 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/afe14e8c7d85447c9755a312a463bc77 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/afe14e8c7d85447c9755a312a463bc77 2023-10-27 23:53:19,585 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/cc065f1761f34463a1fbba299d0df2fc to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf1/cc065f1761f34463a1fbba299d0df2fc 2023-10-27 23:53:19,587 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/35754bb830684c3ea4bf4ae7eec26653 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/35754bb830684c3ea4bf4ae7eec26653 2023-10-27 23:53:19,589 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/3b48fbb9d13b42c3963d26aaac282a36 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/3b48fbb9d13b42c3963d26aaac282a36 2023-10-27 23:53:19,590 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/40ac4a6fb99b46279e5e100d057e6817 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/40ac4a6fb99b46279e5e100d057e6817 2023-10-27 23:53:19,591 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/431c0549e98f4986b0a97348165aabf9 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/431c0549e98f4986b0a97348165aabf9 2023-10-27 23:53:19,592 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/65cd201db5c84287825188b0aaaa3302 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/65cd201db5c84287825188b0aaaa3302 2023-10-27 23:53:19,593 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7811752907234ed6bee370bd314c903e to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7811752907234ed6bee370bd314c903e 2023-10-27 23:53:19,594 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7dfc15023e40482d9bf030d1dd1a1ca8 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/7dfc15023e40482d9bf030d1dd1a1ca8 2023-10-27 23:53:19,595 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/cc4f9d94eccd4b2db7f8b7937dbc5816 2023-10-27 23:53:19,596 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/d134cf340ef84f9e812dd37cf5af4726 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/d134cf340ef84f9e812dd37cf5af4726 2023-10-27 23:53:19,597 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/eb872dc2a7004601b8b23acad657c0f4 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/cf2/eb872dc2a7004601b8b23acad657c0f4 2023-10-27 23:53:19,600 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/recovered.edits/106.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2/recovered.edits/106.seqid 2023-10-27 23:53:19,600 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/3087a119cbcd659122571a2b31e50ec2 2023-10-27 23:53:19,600 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionDeletedRowsDaughter regions 2023-10-27 23:53:19,602 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=84, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:19,604 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionDeletedRowsDaughter from hbase:meta 2023-10-27 23:53:19,606 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionDeletedRowsDaughter' descriptor. 2023-10-27 23:53:19,607 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=84, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:19,607 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionDeletedRowsDaughter' from region states. 2023-10-27 23:53:19,607 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450799607"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:19,608 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450799607"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:19,608 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450799607"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:19,609 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-10-27 23:53:19,610 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3087a119cbcd659122571a2b31e50ec2, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1698450794837.3087a119cbcd659122571a2b31e50ec2.', STARTKEY => '', ENDKEY => ''}, {ENCODED => f0da177e9a77436613e559466a9befb3, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1698450797582.f0da177e9a77436613e559466a9befb3.', STARTKEY => '', ENDKEY => '60'}, {ENCODED => f3419eb3ecb19fbd75b9ef498fab9b83, NAME => 'testSplitTableRegionDeletedRowsDaughter,60,1698450797582.f3419eb3ecb19fbd75b9ef498fab9b83.', STARTKEY => '60', ENDKEY => ''}] 2023-10-27 23:53:19,610 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionDeletedRowsDaughter' as deleted. 2023-10-27 23:53:19,610 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1698450799610"}]},"ts":"9223372036854775807"} 2023-10-27 23:53:19,611 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionDeletedRowsDaughter state from META 2023-10-27 23:53:19,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=84 2023-10-27 23:53:19,714 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=84, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:53:19,715 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 151 msec 2023-10-27 23:53:19,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=84 2023-10-27 23:53:19,869 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionDeletedRowsDaughter, procId: 84 completed 2023-10-27 23:53:19,882 INFO [Listener at localhost/43373] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionDeletedRowsDaughter Thread=415 (was 408) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:55048 [Waiting for operation #48] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:37342 [Waiting for operation #44] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:43588 [Waiting for operation #33] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-metaLookup-shared--pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:58956 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:45746 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:58968 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:43714 [Waiting for operation #36] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:45680 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:45828 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase16:39059-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:37166 [Waiting for operation #50] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-metaLookup-shared--pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-metaLookup-shared--pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:43548 [Waiting for operation #42] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:48708 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:55076 [Waiting for operation #53] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x27f582be-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-metaLookup-shared--pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x23f62efb-metaLookup-shared--pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:45688 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:45730 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1280218889_17 at /127.0.0.1:44476 [Waiting for operation #27] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=862 (was 867), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=337 (was 348), ProcessCount=173 (was 173), AvailableMemoryMB=5499 (was 5523) 2023-10-27 23:53:19,895 INFO [Listener at localhost/43373] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRollbackForSplitTableRegionWithReplica Thread=415, OpenFileDescriptor=862, MaxFileDescriptor=60000, SystemLoadAverage=337, ProcessCount=173, AvailableMemoryMB=5498 2023-10-27 23:53:19,896 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:53:19,911 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-10-27 23:53:19,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(492): Client=jenkins//188.40.62.62 set balanceSwitch=false 2023-10-27 23:53:19,912 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=85, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:53:19,914 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_PRE_OPERATION 2023-10-27 23:53:19,915 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-10-27 23:53:19,916 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:19,917 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a empty. 2023-10-27 23:53:19,917 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:19,917 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackForSplitTableRegionWithReplica regions 2023-10-27 23:53:19,941 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000001 2023-10-27 23:53:19,942 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(7675): creating {ENCODED => b2d34fd5cbfd39a1be2f8cf0f5dbe83a, NAME => 'testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRollbackForSplitTableRegionWithReplica', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp 2023-10-27 23:53:19,953 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(866): Instantiated testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:19,953 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1604): Closing b2d34fd5cbfd39a1be2f8cf0f5dbe83a, disabling compactions & flushes 2023-10-27 23:53:19,953 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1626): Closing region testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:19,953 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:19,953 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1714): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. after waiting 0 ms 2023-10-27 23:53:19,953 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1724): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:19,953 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1838): Closed testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:19,953 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1558): Region close journal for b2d34fd5cbfd39a1be2f8cf0f5dbe83a: 2023-10-27 23:53:19,955 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_ADD_TO_META 2023-10-27 23:53:19,956 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1698450799956"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450799956"}]},"ts":"1698450799956"} 2023-10-27 23:53:19,957 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-10-27 23:53:19,958 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-10-27 23:53:19,958 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450799958"}]},"ts":"1698450799958"} 2023-10-27 23:53:19,959 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=ENABLING in hbase:meta 2023-10-27 23:53:20,011 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-10-27 23:53:20,012 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-27 23:53:20,012 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-10-27 23:53:20,012 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-10-27 23:53:20,012 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-10-27 23:53:20,012 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, ASSIGN}] 2023-10-27 23:53:20,014 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, ASSIGN 2023-10-27 23:53:20,014 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,39059,1698450767147; forceNewPlan=false, retain=false 2023-10-27 23:53:20,165 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-10-27 23:53:20,167 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=86 updating hbase:meta row=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:20,167 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1698450800167"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450800167"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450800167"}]},"ts":"1698450800167"} 2023-10-27 23:53:20,169 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=87, ppid=86, state=RUNNABLE; OpenRegionProcedure b2d34fd5cbfd39a1be2f8cf0f5dbe83a, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:20,320 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:20,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:20,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b2d34fd5cbfd39a1be2f8cf0f5dbe83a, NAME => 'testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:53:20,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:53:20,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:20,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,325 INFO [StoreOpener-b2d34fd5cbfd39a1be2f8cf0f5dbe83a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,326 DEBUG [StoreOpener-b2d34fd5cbfd39a1be2f8cf0f5dbe83a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a/cf1 2023-10-27 23:53:20,326 DEBUG [StoreOpener-b2d34fd5cbfd39a1be2f8cf0f5dbe83a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a/cf1 2023-10-27 23:53:20,327 INFO [StoreOpener-b2d34fd5cbfd39a1be2f8cf0f5dbe83a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b2d34fd5cbfd39a1be2f8cf0f5dbe83a columnFamilyName cf1 2023-10-27 23:53:20,327 INFO [StoreOpener-b2d34fd5cbfd39a1be2f8cf0f5dbe83a-1] regionserver.HStore(310): Store=b2d34fd5cbfd39a1be2f8cf0f5dbe83a/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:20,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-10-27 23:53:20,333 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened b2d34fd5cbfd39a1be2f8cf0f5dbe83a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59875219, jitterRate=-0.10778971016407013}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-10-27 23:53:20,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for b2d34fd5cbfd39a1be2f8cf0f5dbe83a: 2023-10-27 23:53:20,333 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a., pid=87, masterSystemTime=1698450800320 2023-10-27 23:53:20,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:20,335 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:20,335 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=86 updating hbase:meta row=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:20,335 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1698450800335"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450800335"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450800335"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450800335"}]},"ts":"1698450800335"} 2023-10-27 23:53:20,338 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=87, resume processing ppid=86 2023-10-27 23:53:20,338 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=87, ppid=86, state=SUCCESS; OpenRegionProcedure b2d34fd5cbfd39a1be2f8cf0f5dbe83a, server=jenkins-hbase16.apache.org,39059,1698450767147 in 168 msec 2023-10-27 23:53:20,339 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=86, resume processing ppid=85 2023-10-27 23:53:20,339 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=86, ppid=85, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, ASSIGN in 326 msec 2023-10-27 23:53:20,340 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-10-27 23:53:20,340 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450800340"}]},"ts":"1698450800340"} 2023-10-27 23:53:20,341 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=ENABLED in hbase:meta 2023-10-27 23:53:20,353 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_POST_OPERATION 2023-10-27 23:53:20,354 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica in 441 msec 2023-10-27 23:53:20,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$12(2454): Client=jenkins//188.40.62.62 modify table testRollbackForSplitTableRegionWithReplica from 'testRollbackForSplitTableRegionWithReplica', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'testRollbackForSplitTableRegionWithReplica', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '2'}}, {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-10-27 23:53:20,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=88, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:53:20,442 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000002 2023-10-27 23:53:20,443 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000001 2023-10-27 23:53:20,443 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000002 2023-10-27 23:53:20,447 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=89, ppid=88, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=testRollbackForSplitTableRegionWithReplica}] 2023-10-27 23:53:20,462 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, REOPEN/MOVE}] 2023-10-27 23:53:20,464 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, REOPEN/MOVE 2023-10-27 23:53:20,464 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:20,464 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1698450800464"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450800464"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450800464"}]},"ts":"1698450800464"} 2023-10-27 23:53:20,466 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=91, ppid=90, state=RUNNABLE; CloseRegionProcedure b2d34fd5cbfd39a1be2f8cf0f5dbe83a, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:20,617 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:20,618 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing b2d34fd5cbfd39a1be2f8cf0f5dbe83a, disabling compactions & flushes 2023-10-27 23:53:20,619 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:20,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:20,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. after waiting 0 ms 2023-10-27 23:53:20,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:20,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-10-27 23:53:20,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:53:20,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:20,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for b2d34fd5cbfd39a1be2f8cf0f5dbe83a: 2023-10-27 23:53:20,623 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(3510): Not adding moved region record: b2d34fd5cbfd39a1be2f8cf0f5dbe83a to self. 2023-10-27 23:53:20,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,625 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, regionState=CLOSED 2023-10-27 23:53:20,625 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1698450800625"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450800625"}]},"ts":"1698450800625"} 2023-10-27 23:53:20,628 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=91, resume processing ppid=90 2023-10-27 23:53:20,628 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=91, ppid=90, state=SUCCESS; CloseRegionProcedure b2d34fd5cbfd39a1be2f8cf0f5dbe83a, server=jenkins-hbase16.apache.org,39059,1698450767147 in 160 msec 2023-10-27 23:53:20,629 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase16.apache.org,39059,1698450767147; forceNewPlan=false, retain=true 2023-10-27 23:53:20,779 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-10-27 23:53:20,779 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:20,779 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1698450800779"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450800779"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450800779"}]},"ts":"1698450800779"} 2023-10-27 23:53:20,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=92, ppid=90, state=RUNNABLE; OpenRegionProcedure b2d34fd5cbfd39a1be2f8cf0f5dbe83a, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:53:20,932 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:20,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:20,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b2d34fd5cbfd39a1be2f8cf0f5dbe83a, NAME => 'testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:53:20,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:53:20,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:20,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,938 INFO [StoreOpener-b2d34fd5cbfd39a1be2f8cf0f5dbe83a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,938 DEBUG [StoreOpener-b2d34fd5cbfd39a1be2f8cf0f5dbe83a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a/cf1 2023-10-27 23:53:20,939 DEBUG [StoreOpener-b2d34fd5cbfd39a1be2f8cf0f5dbe83a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a/cf1 2023-10-27 23:53:20,939 INFO [StoreOpener-b2d34fd5cbfd39a1be2f8cf0f5dbe83a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b2d34fd5cbfd39a1be2f8cf0f5dbe83a columnFamilyName cf1 2023-10-27 23:53:20,939 INFO [StoreOpener-b2d34fd5cbfd39a1be2f8cf0f5dbe83a-1] regionserver.HStore(310): Store=b2d34fd5cbfd39a1be2f8cf0f5dbe83a/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:53:20,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:53:20,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened b2d34fd5cbfd39a1be2f8cf0f5dbe83a; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61963101, jitterRate=-0.0766778439283371}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-10-27 23:53:20,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for b2d34fd5cbfd39a1be2f8cf0f5dbe83a: 2023-10-27 23:53:20,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a., pid=92, masterSystemTime=1698450800932 2023-10-27 23:53:20,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:20,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:53:20,947 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:53:20,947 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1698450800947"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450800947"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450800947"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450800947"}]},"ts":"1698450800947"} 2023-10-27 23:53:20,950 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=92, resume processing ppid=90 2023-10-27 23:53:20,950 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=92, ppid=90, state=SUCCESS; OpenRegionProcedure b2d34fd5cbfd39a1be2f8cf0f5dbe83a, server=jenkins-hbase16.apache.org,39059,1698450767147 in 167 msec 2023-10-27 23:53:20,951 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=90, resume processing ppid=89 2023-10-27 23:53:20,951 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=90, ppid=89, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, REOPEN/MOVE in 488 msec 2023-10-27 23:53:20,954 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=89, resume processing ppid=88 2023-10-27 23:53:20,954 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=89, ppid=88, state=SUCCESS; ReopenTableRegionsProcedure table=testRollbackForSplitTableRegionWithReplica in 507 msec 2023-10-27 23:53:20,956 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b, ASSIGN}] 2023-10-27 23:53:20,958 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b, ASSIGN 2023-10-27 23:53:20,959 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-10-27 23:53:21,109 DEBUG [jenkins-hbase16:40039] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-10-27 23:53:21,110 DEBUG [jenkins-hbase16:40039] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-27 23:53:21,110 DEBUG [jenkins-hbase16:40039] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-10-27 23:53:21,110 DEBUG [jenkins-hbase16:40039] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-10-27 23:53:21,110 DEBUG [jenkins-hbase16:40039] balancer.BaseLoadBalancer$Cluster(378): Number of tables=3, number of hosts=1, number of racks=1 2023-10-27 23:53:21,112 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=420eeb15f6ee5bc27b6201436916974b, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:21,112 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1698450801112"},{"qualifier":"sn_0001","vlen":46,"tag":[],"timestamp":"1698450801112"},{"qualifier":"state_0001","vlen":7,"tag":[],"timestamp":"1698450801112"}]},"ts":"1698450801112"} 2023-10-27 23:53:21,115 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=94, ppid=93, state=RUNNABLE; OpenRegionProcedure 420eeb15f6ee5bc27b6201436916974b, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:53:21,267 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:53:21,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testRollbackForSplitTableRegionWithReplica,,1698450799912_0001.420eeb15f6ee5bc27b6201436916974b. 2023-10-27 23:53:21,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 420eeb15f6ee5bc27b6201436916974b, NAME => 'testRollbackForSplitTableRegionWithReplica,,1698450799912_0001.420eeb15f6ee5bc27b6201436916974b.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1} 2023-10-27 23:53:21,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:53:21,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica 420eeb15f6ee5bc27b6201436916974b 2023-10-27 23:53:21,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testRollbackForSplitTableRegionWithReplica,,1698450799912_0001.420eeb15f6ee5bc27b6201436916974b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:53:21,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 420eeb15f6ee5bc27b6201436916974b 2023-10-27 23:53:21,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 420eeb15f6ee5bc27b6201436916974b 2023-10-27 23:53:21,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:21,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:21,471 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:21,571 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:21,671 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:21,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:21,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:21,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:22,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:22,172 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:22,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:22,373 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:22,473 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:22,573 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:22,673 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:22,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:22,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:22,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:23,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:23,174 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:23,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:23,375 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:23,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:23,575 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:23,675 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:23,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:23,876 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:23,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:24,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:24,176 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:24,276 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:24,282 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-27 23:53:24,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:24,477 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:24,577 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:24,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:24,777 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:24,878 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:24,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:25,056 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRollbackForSplitTableRegionWithReplica' 2023-10-27 23:53:25,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:25,178 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:25,278 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:25,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:25,479 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:25,579 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:25,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:25,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:25,880 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:25,980 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:26,080 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:26,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:26,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:26,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:26,481 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:26,581 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:26,681 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:26,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:26,882 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:26,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:27,082 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:27,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:27,283 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:27,383 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:27,483 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:27,583 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:27,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:27,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:27,884 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:27,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:28,084 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:28,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:28,285 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:28,385 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:28,485 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:28,585 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:28,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:28,786 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:28,886 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:28,986 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:29,086 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:29,187 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:29,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:29,387 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:29,487 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:29,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:29,688 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:29,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:29,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:29,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:30,088 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:30,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:30,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:30,389 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:30,489 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:30,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:30,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:30,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:30,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:30,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:31,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:31,191 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:31,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:31,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:31,491 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:31,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:31,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:31,792 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:31,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:31,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:32,092 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:32,193 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:32,293 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:32,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:32,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:32,593 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:32,694 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:32,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:32,894 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:32,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:33,094 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:33,195 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:33,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:33,395 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:33,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:33,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:33,696 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:33,796 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:33,896 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:33,996 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:34,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:34,197 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:34,297 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:34,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:34,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:34,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:34,698 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:34,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:34,898 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:34,998 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:35,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:35,199 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:35,299 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:35,399 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:35,499 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:35,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:35,700 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:35,800 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:35,900 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:36,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:36,100 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:36,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:36,301 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:36,401 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:36,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:36,601 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:36,701 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:36,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:36,902 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:37,002 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:37,102 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:37,203 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:37,303 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:37,403 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:37,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:37,603 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:37,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:37,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:37,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:38,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:38,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:38,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:38,305 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:38,405 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:38,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:38,605 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:38,705 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:38,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:38,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:39,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:39,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:39,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:39,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:39,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:39,507 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:39,607 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:39,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:39,808 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:39,908 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:40,008 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:40,108 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:40,208 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:40,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:40,409 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:40,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:40,609 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:40,709 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:40,809 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:40,910 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:41,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:41,110 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:41,210 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:41,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:41,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:41,511 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:41,611 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:41,711 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:41,811 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:41,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:42,012 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:42,112 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:42,212 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:42,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:42,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:42,513 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:42,613 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:42,713 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:42,813 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:42,914 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:43,014 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:43,114 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:43,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:43,314 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:43,414 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:43,515 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:43,615 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:43,715 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:43,815 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:43,915 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:44,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:44,116 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:44,216 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:44,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:44,416 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:44,517 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:44,617 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:44,717 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:44,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:44,917 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:45,018 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:45,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:45,218 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:45,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:45,418 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:45,518 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:45,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:45,719 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:45,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:45,919 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:46,019 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:46,120 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:46,220 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:46,320 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:46,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:46,520 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:46,621 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:46,721 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:46,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:46,921 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:47,021 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:47,122 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:47,222 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:47,322 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:47,422 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:47,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:47,623 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:47,723 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:47,823 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:47,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:48,023 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:48,124 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:48,224 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:48,324 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:48,424 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:48,524 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:48,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:48,725 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:48,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:48,925 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:49,025 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:49,125 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:49,226 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:49,326 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:49,426 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:49,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:49,626 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:49,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:49,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:49,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:50,027 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:50,127 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:50,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:50,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:50,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:50,528 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:50,629 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:50,729 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:50,829 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:50,929 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:51,029 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:51,130 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:51,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:51,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:51,430 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:51,530 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:51,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:51,731 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:51,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:51,931 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:52,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:52,132 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:52,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:52,332 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:52,432 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:52,532 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:52,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:52,733 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:52,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:52,933 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:53,033 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:53,134 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:53,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:53,334 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:53,434 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:53,535 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:53,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:53,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:53,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:53,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:54,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:54,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:54,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:54,336 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:54,436 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:54,537 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:54,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:54,737 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:54,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:54,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:55,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:55,138 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:55,238 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:55,338 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:55,438 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:55,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:55,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:55,739 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:55,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:55,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:56,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:56,140 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:56,240 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:56,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:56,440 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:56,541 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:56,641 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:56,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:56,841 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:56,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:57,042 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:57,142 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:57,242 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:57,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:57,442 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:57,543 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:57,643 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:57,743 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:57,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:57,943 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:58,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:58,144 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:58,244 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:58,344 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:58,444 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:58,545 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:58,645 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:58,745 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:58,845 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:58,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:59,046 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:59,146 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:59,246 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:59,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:59,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:59,547 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:59,647 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:59,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:59,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:53:59,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:00,048 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:00,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:00,248 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:00,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:00,449 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:00,549 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:00,649 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:00,749 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:00,849 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:00,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:01,050 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:01,150 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:01,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:01,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:01,451 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:01,551 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:01,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:01,751 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:01,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:01,952 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:02,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:02,152 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:02,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:02,352 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:02,453 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:02,553 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:02,653 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:02,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:02,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:02,954 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:03,054 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:03,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:03,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:03,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:03,455 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:03,555 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:03,655 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:03,755 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:03,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:03,956 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:04,056 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:04,156 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:04,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:04,356 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:04,457 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:04,557 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:04,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:04,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:04,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:04,957 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:05,058 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:05,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:05,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:05,358 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:05,458 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:05,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:05,659 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:05,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:05,859 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:05,959 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:06,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:06,160 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:06,260 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:06,360 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:06,461 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:06,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:06,661 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:06,761 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:06,861 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:06,962 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:07,062 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:07,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:07,262 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:07,362 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:07,463 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:07,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:07,663 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:07,763 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:07,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:07,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:08,064 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:08,164 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:08,264 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:08,364 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:08,464 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:08,565 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:08,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:08,765 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:08,865 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:08,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:09,066 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:09,166 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:09,266 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:09,366 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:09,466 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:09,567 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:09,667 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:09,767 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:09,867 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:09,967 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:10,068 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:10,168 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:10,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:10,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:10,468 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:10,569 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:10,669 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:10,769 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:10,869 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:10,970 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:11,070 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:11,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:11,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:11,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:11,471 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:11,571 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:11,671 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:11,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:11,871 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:11,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:12,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:12,172 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:12,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:12,372 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:12,473 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:12,573 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:12,673 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:12,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:12,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:12,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:13,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:13,174 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:13,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:13,374 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:13,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:13,575 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:13,675 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:13,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:13,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:13,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:14,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:14,176 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:14,276 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:14,376 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:14,477 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:14,577 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:14,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:14,777 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:14,878 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:14,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:15,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:15,178 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:15,278 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:15,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:15,479 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:15,579 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:15,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:15,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:15,880 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:15,980 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:16,080 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:16,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:16,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:16,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:16,481 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:16,581 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:16,681 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:16,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:16,882 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:16,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:17,082 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:17,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:17,283 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:17,383 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:17,483 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:17,583 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:17,683 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:17,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:17,884 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:17,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:18,084 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:18,184 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:18,285 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:18,385 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:18,485 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:18,585 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:18,685 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:18,786 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:18,886 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:18,986 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:19,086 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:19,187 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:19,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:19,387 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:19,487 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:19,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:19,688 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:19,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:19,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:19,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:20,088 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:20,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:20,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:20,389 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:20,489 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:20,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=88 2023-10-27 23:54:20,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:20,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=88 2023-10-27 23:54:20,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:20,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:20,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=88 2023-10-27 23:54:20,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:20,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:21,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:21,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=88 2023-10-27 23:54:21,191 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:21,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:21,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:21,491 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:21,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:21,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=88 2023-10-27 23:54:21,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:21,792 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:21,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:21,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:22,092 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:22,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:22,293 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:22,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:22,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:22,593 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:22,642 DEBUG [Listener at localhost/43373] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:22,643 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:22,644 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:22,644 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:22,644 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:22,646 ERROR [PEWorker-1] assignment.SplitTableRegionProcedure(347): Splitting b2d34fd5cbfd39a1be2f8cf0f5dbe83a, pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up at org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.createUnassignProcedures(SplitTableRegionProcedure.java:881) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:298) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:94) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1645) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:54:22,648 INFO [PEWorker-1] procedure2.ProcedureExecutor(1527): Rolled back pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 exec-time=6 msec 2023-10-27 23:54:22,694 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:22,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-10-27 23:54:22,894 INFO [StoreOpener-420eeb15f6ee5bc27b6201436916974b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 420eeb15f6ee5bc27b6201436916974b 2023-10-27 23:54:22,897 DEBUG [StoreOpener-420eeb15f6ee5bc27b6201436916974b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a/cf1 2023-10-27 23:54:22,897 DEBUG [StoreOpener-420eeb15f6ee5bc27b6201436916974b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a/cf1 2023-10-27 23:54:22,897 INFO [StoreOpener-420eeb15f6ee5bc27b6201436916974b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 420eeb15f6ee5bc27b6201436916974b columnFamilyName cf1 2023-10-27 23:54:22,898 INFO [StoreOpener-420eeb15f6ee5bc27b6201436916974b-1] regionserver.HStore(310): Store=420eeb15f6ee5bc27b6201436916974b/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:22,899 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 420eeb15f6ee5bc27b6201436916974b; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74941828, jitterRate=0.11672025918960571}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-10-27 23:54:22,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 420eeb15f6ee5bc27b6201436916974b: 2023-10-27 23:54:22,900 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1698450799912_0001.420eeb15f6ee5bc27b6201436916974b., pid=94, masterSystemTime=1698450801267 2023-10-27 23:54:22,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1698450799912_0001.420eeb15f6ee5bc27b6201436916974b. 2023-10-27 23:54:22,901 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testRollbackForSplitTableRegionWithReplica,,1698450799912_0001.420eeb15f6ee5bc27b6201436916974b. 2023-10-27 23:54:22,902 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=420eeb15f6ee5bc27b6201436916974b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:22,902 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1698450862901"},{"qualifier":"server_0001","vlen":32,"tag":[],"timestamp":"1698450862901"},{"qualifier":"serverstartcode_0001","vlen":8,"tag":[],"timestamp":"1698450862901"},{"qualifier":"seqnumDuringOpen_0001","vlen":8,"tag":[],"timestamp":"1698450862901"}]},"ts":"1698450862901"} 2023-10-27 23:54:22,905 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=94, resume processing ppid=93 2023-10-27 23:54:22,905 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=94, ppid=93, state=SUCCESS; OpenRegionProcedure 420eeb15f6ee5bc27b6201436916974b, server=jenkins-hbase16.apache.org,43641,1698450767195 in 1 mins, 1.79 sec 2023-10-27 23:54:22,906 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=93, resume processing ppid=88 2023-10-27 23:54:22,906 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=93, ppid=88, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b, ASSIGN in 1 mins, 1.949 sec 2023-10-27 23:54:22,908 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica in 1 mins, 2.483 sec 2023-10-27 23:54:22,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] ipc.RpcServer(528): (responseTooSlow): {"call":"ModifyTable(org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$ModifyTableRequest)","starttimems":"1698450800419","responsesize":"2","method":"ModifyTable","param":"TODO: class org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$ModifyTableRequest","processingtimems":62489,"client":"188.40.62.62:59288","queuetimems":0,"class":"HMaster"} 2023-10-27 23:54:22,909 DEBUG [RPCClient-NioEventLoopGroup-6-2] ipc.NettyRpcDuplexHandler(187): Unknown callId: 611, skipping over this response of 3 bytes 2023-10-27 23:54:24,894 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(335): procId=95 exception: org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up 2023-10-27 23:54:24,895 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:54:24,897 INFO [Listener at localhost/43373] client.HBaseAdmin$15(890): Started disable of testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:24,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:24,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=96, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:24,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=96 2023-10-27 23:54:24,900 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450864900"}]},"ts":"1698450864900"} 2023-10-27 23:54:24,901 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=DISABLING in hbase:meta 2023-10-27 23:54:24,933 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRollbackForSplitTableRegionWithReplica to state=DISABLING 2023-10-27 23:54:24,933 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=97, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, UNASSIGN}, {pid=98, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b, UNASSIGN}] 2023-10-27 23:54:24,934 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b, UNASSIGN 2023-10-27 23:54:24,935 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=420eeb15f6ee5bc27b6201436916974b, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:24,935 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1698450864935"},{"qualifier":"sn_0001","vlen":46,"tag":[],"timestamp":"1698450864935"},{"qualifier":"state_0001","vlen":7,"tag":[],"timestamp":"1698450864935"}]},"ts":"1698450864935"} 2023-10-27 23:54:24,936 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=99, ppid=98, state=RUNNABLE; CloseRegionProcedure 420eeb15f6ee5bc27b6201436916974b, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:54:24,938 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, UNASSIGN 2023-10-27 23:54:24,939 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:24,939 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1698450864939"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450864939"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450864939"}]},"ts":"1698450864939"} 2023-10-27 23:54:24,940 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=100, ppid=97, state=RUNNABLE; CloseRegionProcedure b2d34fd5cbfd39a1be2f8cf0f5dbe83a, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:54:25,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=96 2023-10-27 23:54:25,088 DEBUG [RSProcedureDispatcher-pool-3] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:25,089 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 420eeb15f6ee5bc27b6201436916974b 2023-10-27 23:54:25,089 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 420eeb15f6ee5bc27b6201436916974b, disabling compactions & flushes 2023-10-27 23:54:25,089 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testRollbackForSplitTableRegionWithReplica,,1698450799912_0001.420eeb15f6ee5bc27b6201436916974b. 2023-10-27 23:54:25,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1698450799912_0001.420eeb15f6ee5bc27b6201436916974b. 2023-10-27 23:54:25,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1698450799912_0001.420eeb15f6ee5bc27b6201436916974b. after waiting 0 ms 2023-10-27 23:54:25,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1698450799912_0001.420eeb15f6ee5bc27b6201436916974b. 2023-10-27 23:54:25,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:25,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testRollbackForSplitTableRegionWithReplica,,1698450799912_0001.420eeb15f6ee5bc27b6201436916974b. 2023-10-27 23:54:25,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 420eeb15f6ee5bc27b6201436916974b: 2023-10-27 23:54:25,091 DEBUG [RSProcedureDispatcher-pool-4] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,092 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:54:25,092 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing b2d34fd5cbfd39a1be2f8cf0f5dbe83a, disabling compactions & flushes 2023-10-27 23:54:25,093 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:54:25,093 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:54:25,093 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. after waiting 0 ms 2023-10-27 23:54:25,093 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:54:25,093 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 420eeb15f6ee5bc27b6201436916974b 2023-10-27 23:54:25,094 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=420eeb15f6ee5bc27b6201436916974b, regionState=CLOSED 2023-10-27 23:54:25,095 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1698450865094"},{"qualifier":"state_0001","vlen":6,"tag":[],"timestamp":"1698450865094"}]},"ts":"1698450865094"} 2023-10-27 23:54:25,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-10-27 23:54:25,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:25,099 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a. 2023-10-27 23:54:25,099 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for b2d34fd5cbfd39a1be2f8cf0f5dbe83a: 2023-10-27 23:54:25,100 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=99, resume processing ppid=98 2023-10-27 23:54:25,100 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=99, ppid=98, state=SUCCESS; CloseRegionProcedure 420eeb15f6ee5bc27b6201436916974b, server=jenkins-hbase16.apache.org,43641,1698450767195 in 161 msec 2023-10-27 23:54:25,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:54:25,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=98, ppid=96, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b, UNASSIGN in 167 msec 2023-10-27 23:54:25,101 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, regionState=CLOSED 2023-10-27 23:54:25,101 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1698450865101"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450865101"}]},"ts":"1698450865101"} 2023-10-27 23:54:25,104 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=100, resume processing ppid=97 2023-10-27 23:54:25,104 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=100, ppid=97, state=SUCCESS; CloseRegionProcedure b2d34fd5cbfd39a1be2f8cf0f5dbe83a, server=jenkins-hbase16.apache.org,39059,1698450767147 in 162 msec 2023-10-27 23:54:25,105 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=97, resume processing ppid=96 2023-10-27 23:54:25,106 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=97, ppid=96, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, UNASSIGN in 171 msec 2023-10-27 23:54:25,106 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450865106"}]},"ts":"1698450865106"} 2023-10-27 23:54:25,107 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=DISABLED in hbase:meta 2023-10-27 23:54:25,116 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRollbackForSplitTableRegionWithReplica to state=DISABLED 2023-10-27 23:54:25,118 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica in 220 msec 2023-10-27 23:54:25,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=96 2023-10-27 23:54:25,203 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRollbackForSplitTableRegionWithReplica, procId: 96 completed 2023-10-27 23:54:25,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:25,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=101, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:25,206 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=101, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:25,207 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=101, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:25,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=101 2023-10-27 23:54:25,211 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:54:25,213 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a/recovered.edits] 2023-10-27 23:54:25,217 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a/recovered.edits/7.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a/recovered.edits/7.seqid 2023-10-27 23:54:25,217 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/b2d34fd5cbfd39a1be2f8cf0f5dbe83a 2023-10-27 23:54:25,217 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackForSplitTableRegionWithReplica regions 2023-10-27 23:54:25,219 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=101, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:25,221 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testRollbackForSplitTableRegionWithReplica from hbase:meta 2023-10-27 23:54:25,223 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRollbackForSplitTableRegionWithReplica' descriptor. 2023-10-27 23:54:25,224 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=101, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:25,224 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRollbackForSplitTableRegionWithReplica' from region states. 2023-10-27 23:54:25,224 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450865224"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:25,224 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica,,1698450799912_0001.420eeb15f6ee5bc27b6201436916974b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450865224"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:25,226 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 2 regions from META 2023-10-27 23:54:25,226 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => b2d34fd5cbfd39a1be2f8cf0f5dbe83a, NAME => 'testRollbackForSplitTableRegionWithReplica,,1698450799912.b2d34fd5cbfd39a1be2f8cf0f5dbe83a.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 420eeb15f6ee5bc27b6201436916974b, NAME => 'testRollbackForSplitTableRegionWithReplica,,1698450799912_0001.420eeb15f6ee5bc27b6201436916974b.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1}] 2023-10-27 23:54:25,226 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRollbackForSplitTableRegionWithReplica' as deleted. 2023-10-27 23:54:25,226 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1698450865226"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:25,228 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRollbackForSplitTableRegionWithReplica state from META 2023-10-27 23:54:25,239 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=101, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:25,240 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica in 35 msec 2023-10-27 23:54:25,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=101 2023-10-27 23:54:25,309 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRollbackForSplitTableRegionWithReplica, procId: 101 completed 2023-10-27 23:54:25,320 INFO [Listener at localhost/43373] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRollbackForSplitTableRegionWithReplica Thread=350 (was 415), OpenFileDescriptor=658 (was 862), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=225 (was 337), ProcessCount=171 (was 173), AvailableMemoryMB=5483 (was 5498) 2023-10-27 23:54:25,333 INFO [Listener at localhost/43373] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testInvalidSplitKey Thread=351, OpenFileDescriptor=658, MaxFileDescriptor=60000, SystemLoadAverage=225, ProcessCount=171, AvailableMemoryMB=5483 2023-10-27 23:54:25,333 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:54:25,341 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-10-27 23:54:25,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(492): Client=jenkins//188.40.62.62 set balanceSwitch=false 2023-10-27 23:54:25,343 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=102, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:25,345 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_PRE_OPERATION 2023-10-27 23:54:25,346 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-10-27 23:54:25,347 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:25,348 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd empty. 2023-10-27 23:54:25,348 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:25,348 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testInvalidSplitKey regions 2023-10-27 23:54:25,360 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/.tabledesc/.tableinfo.0000000001 2023-10-27 23:54:25,361 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8f308088886703aa0a6843bebc9038fd, NAME => 'testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testInvalidSplitKey', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp 2023-10-27 23:54:25,369 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(866): Instantiated testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:25,369 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1604): Closing 8f308088886703aa0a6843bebc9038fd, disabling compactions & flushes 2023-10-27 23:54:25,369 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1626): Closing region testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:25,369 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:25,369 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1714): Acquired close lock on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. after waiting 0 ms 2023-10-27 23:54:25,369 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1724): Updates disabled for region testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:25,369 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1838): Closed testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:25,369 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1558): Region close journal for 8f308088886703aa0a6843bebc9038fd: 2023-10-27 23:54:25,371 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_ADD_TO_META 2023-10-27 23:54:25,372 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1698450865372"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450865372"}]},"ts":"1698450865372"} 2023-10-27 23:54:25,373 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-10-27 23:54:25,374 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-10-27 23:54:25,374 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450865374"}]},"ts":"1698450865374"} 2023-10-27 23:54:25,375 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=ENABLING in hbase:meta 2023-10-27 23:54:25,391 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-10-27 23:54:25,392 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-27 23:54:25,392 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-10-27 23:54:25,392 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-10-27 23:54:25,392 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-10-27 23:54:25,392 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testInvalidSplitKey, region=8f308088886703aa0a6843bebc9038fd, ASSIGN}] 2023-10-27 23:54:25,394 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testInvalidSplitKey, region=8f308088886703aa0a6843bebc9038fd, ASSIGN 2023-10-27 23:54:25,394 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testInvalidSplitKey, region=8f308088886703aa0a6843bebc9038fd, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,39059,1698450767147; forceNewPlan=false, retain=false 2023-10-27 23:54:25,545 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-10-27 23:54:25,546 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=103 updating hbase:meta row=8f308088886703aa0a6843bebc9038fd, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,546 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1698450865546"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450865546"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450865546"}]},"ts":"1698450865546"} 2023-10-27 23:54:25,548 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=104, ppid=103, state=RUNNABLE; OpenRegionProcedure 8f308088886703aa0a6843bebc9038fd, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:54:25,699 DEBUG [RSProcedureDispatcher-pool-5] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:25,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8f308088886703aa0a6843bebc9038fd, NAME => 'testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:54:25,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:54:25,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testInvalidSplitKey 8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:25,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:25,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:25,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:25,706 INFO [StoreOpener-8f308088886703aa0a6843bebc9038fd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:25,707 DEBUG [StoreOpener-8f308088886703aa0a6843bebc9038fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1 2023-10-27 23:54:25,707 DEBUG [StoreOpener-8f308088886703aa0a6843bebc9038fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1 2023-10-27 23:54:25,708 INFO [StoreOpener-8f308088886703aa0a6843bebc9038fd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8f308088886703aa0a6843bebc9038fd columnFamilyName cf1 2023-10-27 23:54:25,708 INFO [StoreOpener-8f308088886703aa0a6843bebc9038fd-1] regionserver.HStore(310): Store=8f308088886703aa0a6843bebc9038fd/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:25,708 INFO [StoreOpener-8f308088886703aa0a6843bebc9038fd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:25,710 DEBUG [StoreOpener-8f308088886703aa0a6843bebc9038fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2 2023-10-27 23:54:25,710 DEBUG [StoreOpener-8f308088886703aa0a6843bebc9038fd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2 2023-10-27 23:54:25,710 INFO [StoreOpener-8f308088886703aa0a6843bebc9038fd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8f308088886703aa0a6843bebc9038fd columnFamilyName cf2 2023-10-27 23:54:25,711 INFO [StoreOpener-8f308088886703aa0a6843bebc9038fd-1] regionserver.HStore(310): Store=8f308088886703aa0a6843bebc9038fd/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:25,712 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:25,712 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:25,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testInvalidSplitKey descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:54:25,715 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:25,717 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-10-27 23:54:25,717 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 8f308088886703aa0a6843bebc9038fd; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65523693, jitterRate=-0.023620888590812683}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:54:25,717 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 8f308088886703aa0a6843bebc9038fd: 2023-10-27 23:54:25,718 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd., pid=104, masterSystemTime=1698450865699 2023-10-27 23:54:25,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:25,720 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:25,720 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=103 updating hbase:meta row=8f308088886703aa0a6843bebc9038fd, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,720 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1698450865720"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450865720"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450865720"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450865720"}]},"ts":"1698450865720"} 2023-10-27 23:54:25,724 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=104, resume processing ppid=103 2023-10-27 23:54:25,724 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=104, ppid=103, state=SUCCESS; OpenRegionProcedure 8f308088886703aa0a6843bebc9038fd, server=jenkins-hbase16.apache.org,39059,1698450767147 in 175 msec 2023-10-27 23:54:25,725 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=103, resume processing ppid=102 2023-10-27 23:54:25,726 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=103, ppid=102, state=SUCCESS; TransitRegionStateProcedure table=testInvalidSplitKey, region=8f308088886703aa0a6843bebc9038fd, ASSIGN in 332 msec 2023-10-27 23:54:25,726 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-10-27 23:54:25,726 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450865726"}]},"ts":"1698450865726"} 2023-10-27 23:54:25,728 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=ENABLED in hbase:meta 2023-10-27 23:54:25,764 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_POST_OPERATION 2023-10-27 23:54:25,766 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey in 422 msec 2023-10-27 23:54:25,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:25,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-10-27 23:54:25,863 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-10-27 23:54:25,863 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:25,863 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-10-27 23:54:25,863 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-10-27 23:54:25,863 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:25,864 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:25,871 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,871 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:25,871 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:25,871 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:25,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:25,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:25,872 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,872 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:25,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:25,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:25,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:25,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:25,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:25,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:25,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:25,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:25,873 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:25,873 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:25,873 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:25,873 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:25,873 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:25,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:25,879 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:25,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:25,879 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:25,879 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:25,879 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:25,879 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:25,879 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:25,879 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:25,879 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:25,879 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:25,880 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:25,880 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:25,880 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. started... 2023-10-27 23:54:25,882 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f308088886703aa0a6843bebc9038fd 2/2 column families, dataSize=120 B heapSize=928 B 2023-10-27 23:54:25,883 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,891 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,891 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,891 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:25,891 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,891 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:25,900 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/6cca79f962be4277a64d88f342e41d36 2023-10-27 23:54:25,942 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/4103850a0cc84fe5ae825944ee6ace84 2023-10-27 23:54:25,946 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/6cca79f962be4277a64d88f342e41d36 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/6cca79f962be4277a64d88f342e41d36 2023-10-27 23:54:25,950 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/6cca79f962be4277a64d88f342e41d36, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:54:25,951 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/4103850a0cc84fe5ae825944ee6ace84 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/4103850a0cc84fe5ae825944ee6ace84 2023-10-27 23:54:25,955 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/4103850a0cc84fe5ae825944ee6ace84, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:54:25,956 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 8f308088886703aa0a6843bebc9038fd in 74ms, sequenceid=6, compaction requested=false 2023-10-27 23:54:25,956 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testInvalidSplitKey' 2023-10-27 23:54:25,956 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f308088886703aa0a6843bebc9038fd: 2023-10-27 23:54:25,956 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:25,956 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:25,956 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:25,956 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,956 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:25,956 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:25,966 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,966 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:25,966 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,966 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:25,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-10-27 23:54:25,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b95c5a4[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:25,969 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-10-27 23:54:25,969 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,974 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,974 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,974 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,975 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:25,975 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:25,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:25,975 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:25,975 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:25,975 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,975 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:25,975 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:25,975 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:25,975 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:25,975 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:25,975 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:25,983 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:25,983 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:25,983 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:25,991 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:25,991 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:25,991 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:25,991 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,991 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:25,991 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:25,991 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:54:25,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-10-27 23:54:25,995 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-10-27 23:54:25,995 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:25,995 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:25,995 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-10-27 23:54:25,995 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:26,005 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,005 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,005 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,005 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,005 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,005 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,005 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:26,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,005 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,005 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,006 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,007 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,007 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,007 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,016 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,016 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,016 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,016 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-10-27 23:54:26,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:26,016 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,016 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,016 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:26,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,016 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,017 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:26,017 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,016 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,017 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,016 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,017 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,017 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,017 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,017 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,018 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,018 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,018 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,117 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:26,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:26,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:26,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:26,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-10-27 23:54:26,130 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-10-27 23:54:26,130 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:26,130 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-10-27 23:54:26,130 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-10-27 23:54:26,130 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,130 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,158 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,158 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,158 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,158 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,159 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,159 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:26,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:26,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:26,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:26,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:26,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:26,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:26,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:26,160 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:26,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:26,160 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:26,160 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:26,160 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:26,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:26,160 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:26,160 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:26,160 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:26,160 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:26,160 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:26,160 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:26,160 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:26,160 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:26,160 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:26,161 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:26,161 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:26,161 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:26,161 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. started... 2023-10-27 23:54:26,161 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f308088886703aa0a6843bebc9038fd 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:26,169 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/f6600f6e12724573ac25d59aaa99f951 2023-10-27 23:54:26,171 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,178 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/17f319a4fe88455a9585ab4df0ba18b5 2023-10-27 23:54:26,179 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,179 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,180 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:26,180 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,180 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:26,182 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/f6600f6e12724573ac25d59aaa99f951 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/f6600f6e12724573ac25d59aaa99f951 2023-10-27 23:54:26,186 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/f6600f6e12724573ac25d59aaa99f951, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:54:26,187 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/17f319a4fe88455a9585ab4df0ba18b5 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/17f319a4fe88455a9585ab4df0ba18b5 2023-10-27 23:54:26,192 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/17f319a4fe88455a9585ab4df0ba18b5, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:54:26,193 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 8f308088886703aa0a6843bebc9038fd in 32ms, sequenceid=19, compaction requested=false 2023-10-27 23:54:26,193 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f308088886703aa0a6843bebc9038fd: 2023-10-27 23:54:26,193 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:26,193 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:26,193 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:26,193 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,193 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:26,193 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:26,199 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,199 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:26,200 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,200 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:26,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-10-27 23:54:26,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@305faa2f[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:26,201 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-10-27 23:54:26,201 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,208 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,208 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,208 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,208 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:26,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,208 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:26,208 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:26,208 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:26,208 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,208 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:26,208 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,208 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:26,208 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:26,208 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:26,208 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:26,216 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:26,216 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:26,216 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:26,224 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:26,224 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:26,224 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:26,233 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:26,233 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,233 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:26,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:26,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:26,233 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:26,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:54:26,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-10-27 23:54:26,236 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-10-27 23:54:26,236 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:26,236 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:26,236 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-10-27 23:54:26,236 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:26,246 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,246 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,246 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,246 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,246 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,246 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,246 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:26,247 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:26,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,247 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,248 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,249 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:26,249 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,249 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,258 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:26,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-10-27 23:54:26,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:26,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,259 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:26,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,259 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,258 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,259 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,359 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:26,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:26,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:26,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:26,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-10-27 23:54:26,372 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-10-27 23:54:26,372 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:26,372 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-10-27 23:54:26,372 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-10-27 23:54:26,373 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,373 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,383 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,383 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,383 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,383 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,383 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,383 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:26,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:26,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:26,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:26,384 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:26,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:26,384 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:26,384 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:26,384 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:26,384 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:26,384 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:26,384 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:26,385 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:26,385 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:26,385 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:26,385 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:26,385 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:26,385 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:26,385 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:26,385 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:26,385 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:26,385 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. started... 2023-10-27 23:54:26,385 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f308088886703aa0a6843bebc9038fd 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:26,391 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,391 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,391 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,391 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,391 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:26,391 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:26,393 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/d98880e3ea30496d94be2a0407539c8c 2023-10-27 23:54:26,403 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/5dbc00e8b1084e1cbc0099c25976383f 2023-10-27 23:54:26,408 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/d98880e3ea30496d94be2a0407539c8c as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/d98880e3ea30496d94be2a0407539c8c 2023-10-27 23:54:26,411 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/d98880e3ea30496d94be2a0407539c8c, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:54:26,412 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/5dbc00e8b1084e1cbc0099c25976383f as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/5dbc00e8b1084e1cbc0099c25976383f 2023-10-27 23:54:26,415 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/5dbc00e8b1084e1cbc0099c25976383f, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:54:26,416 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 8f308088886703aa0a6843bebc9038fd in 31ms, sequenceid=32, compaction requested=true 2023-10-27 23:54:26,416 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f308088886703aa0a6843bebc9038fd: 2023-10-27 23:54:26,416 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:26,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:26,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:26,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:26,416 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:26,424 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,424 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:26,425 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,425 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:26,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-10-27 23:54:26,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42130d81[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:26,427 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-10-27 23:54:26,427 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,433 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,433 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,433 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,433 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:26,433 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:26,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,433 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:26,433 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:26,433 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:26,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,433 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,433 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:26,433 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:26,433 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:26,433 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:26,433 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:26,441 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:26,441 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:26,441 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:26,449 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:26,449 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:26,449 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:26,458 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:26,458 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,458 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:26,458 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:54:26,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-10-27 23:54:26,461 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-10-27 23:54:26,461 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:26,461 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:26,461 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-10-27 23:54:26,461 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:26,471 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,471 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,471 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,472 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,471 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:26,471 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,472 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,472 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,472 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,473 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,474 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:26,474 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,474 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,501 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,501 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,501 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,501 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:26,501 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,501 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,501 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,501 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,501 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-10-27 23:54:26,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:26,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,502 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,502 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,502 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:26,502 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,502 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,502 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,502 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,502 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,502 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,502 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,502 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,602 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:26,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:26,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:26,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:26,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-10-27 23:54:26,616 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-10-27 23:54:26,616 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:26,616 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-10-27 23:54:26,616 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-10-27 23:54:26,616 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,616 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,624 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,624 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,624 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,624 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,625 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,625 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:26,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:26,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:26,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:26,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:26,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:26,626 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:26,626 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:26,626 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:26,626 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:26,626 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:26,626 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:26,626 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:26,626 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:26,626 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:26,626 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:26,626 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:26,626 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:26,626 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:26,627 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. started... 2023-10-27 23:54:26,626 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:26,627 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:26,627 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:26,627 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f308088886703aa0a6843bebc9038fd 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:26,633 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,633 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,633 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,633 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:26,633 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,633 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:26,634 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/fc7761a8754748359e67682a0f9f5891 2023-10-27 23:54:26,662 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/9da1dbbb713f45a3be98ce38907c3526 2023-10-27 23:54:26,665 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/fc7761a8754748359e67682a0f9f5891 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/fc7761a8754748359e67682a0f9f5891 2023-10-27 23:54:26,669 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/fc7761a8754748359e67682a0f9f5891, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:54:26,670 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/9da1dbbb713f45a3be98ce38907c3526 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/9da1dbbb713f45a3be98ce38907c3526 2023-10-27 23:54:26,674 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/9da1dbbb713f45a3be98ce38907c3526, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:54:26,674 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 8f308088886703aa0a6843bebc9038fd in 47ms, sequenceid=45, compaction requested=true 2023-10-27 23:54:26,675 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f308088886703aa0a6843bebc9038fd: 2023-10-27 23:54:26,675 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:26,675 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:26,675 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:26,675 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,675 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:26,675 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:26,805 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,805 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:26,806 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,806 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:26,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-10-27 23:54:26,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56ab07fe[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:26,808 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-10-27 23:54:26,808 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,904 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,904 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,904 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,904 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,904 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,904 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:26,904 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:26,904 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:26,904 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:26,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,904 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:26,905 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,905 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:26,905 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:26,905 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:26,916 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:26,916 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:26,916 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:26,924 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:26,924 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,924 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:26,925 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:26,933 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:26,933 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:26,933 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:54:26,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-10-27 23:54:26,936 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-10-27 23:54:26,936 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:26,936 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:26,937 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-10-27 23:54:26,937 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:26,946 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,946 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,946 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,946 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,946 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,946 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,946 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,947 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:26,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,947 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:26,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,947 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,949 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:26,949 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:26,949 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,958 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,958 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,958 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,958 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:26,958 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:26,958 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-10-27 23:54:26,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:26,958 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,958 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,958 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,958 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:26,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:26,958 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:26,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:26,958 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:26,958 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,959 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:26,959 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:26,959 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,959 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:26,959 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:26,959 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:26,959 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,059 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:27,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:27,068 ERROR [BP-446492043-188.40.62.62-1698450761445 heartbeating to localhost/127.0.0.1:46525] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-446492043-188.40.62.62-1698450761445 (Datanode Uuid 9d32eec7-ccca-436f-a2e6-4184d95cd961) service to localhost/127.0.0.1:46525 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-10-27 23:54:27,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:27,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:27,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-10-27 23:54:27,072 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-10-27 23:54:27,072 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:27,072 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-10-27 23:54:27,072 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-10-27 23:54:27,073 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,073 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,083 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:27,083 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:27,083 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:27,083 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:27,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:27,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:27,083 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,083 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:27,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:27,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:27,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:27,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:27,084 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:27,084 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:27,084 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:27,084 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:27,084 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:27,084 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:27,085 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:27,085 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:27,085 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:27,085 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:27,085 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:27,085 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:27,085 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:27,085 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:27,085 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:27,085 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:27,085 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. started... 2023-10-27 23:54:27,085 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f308088886703aa0a6843bebc9038fd 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:27,091 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,091 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,091 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,091 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,091 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:27,091 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:27,092 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/42750e8b000145bc9fc23356929d3fbd 2023-10-27 23:54:27,102 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/3925004ad5814455993415b083261b8a 2023-10-27 23:54:27,106 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/42750e8b000145bc9fc23356929d3fbd as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/42750e8b000145bc9fc23356929d3fbd 2023-10-27 23:54:27,109 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/42750e8b000145bc9fc23356929d3fbd, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:54:27,110 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/3925004ad5814455993415b083261b8a as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/3925004ad5814455993415b083261b8a 2023-10-27 23:54:27,113 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/3925004ad5814455993415b083261b8a, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:54:27,114 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 8f308088886703aa0a6843bebc9038fd in 29ms, sequenceid=58, compaction requested=true 2023-10-27 23:54:27,114 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f308088886703aa0a6843bebc9038fd: 2023-10-27 23:54:27,114 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:27,114 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:27,114 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:27,114 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,114 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:27,114 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:27,144 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,144 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:27,145 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,145 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-10-27 23:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23e6506a[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:27,147 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-10-27 23:54:27,147 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,178 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,178 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,178 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,178 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:27,178 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:27,178 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:27,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,178 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,179 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:27,179 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:27,179 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:27,179 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:27,179 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:27,179 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:27,179 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:27,191 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:27,191 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:27,191 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:27,199 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:27,199 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,199 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:27,199 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:27,200 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:27,199 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:27,200 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:27,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:27,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:54:27,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-10-27 23:54:27,203 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-10-27 23:54:27,203 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:27,203 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:27,203 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-10-27 23:54:27,203 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:27,213 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,213 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,213 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,213 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,213 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,213 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,213 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,213 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:27,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:27,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:27,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:27,214 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,214 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:27,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:27,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:27,216 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:27,216 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,216 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:27,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:27,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:27,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:27,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-10-27 23:54:27,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:27,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:27,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:27,224 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:27,225 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:27,225 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:27,225 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:27,225 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:27,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:27,225 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:27,225 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:27,225 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:27,225 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,226 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,226 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,226 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:27,226 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:27,226 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,325 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:27,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:27,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:27,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:27,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-10-27 23:54:27,338 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-10-27 23:54:27,338 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:27,338 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-10-27 23:54:27,338 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-10-27 23:54:27,338 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,338 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,346 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,346 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:27,346 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:27,346 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:27,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:27,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:27,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:27,347 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,347 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:27,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-10-27 23:54:27,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:27,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:27,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:27,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:27,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:27,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:27,348 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:27,348 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:27,349 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. started... 2023-10-27 23:54:27,349 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8f308088886703aa0a6843bebc9038fd 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:27,356 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/9c9c41281b6e407897ffe995bbcef480 2023-10-27 23:54:27,357 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,357 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,358 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,358 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,358 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:27,358 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:27,366 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/67f9d0e4feaf4b538606a6c019d1ae07 2023-10-27 23:54:27,370 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/9c9c41281b6e407897ffe995bbcef480 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/9c9c41281b6e407897ffe995bbcef480 2023-10-27 23:54:27,374 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/9c9c41281b6e407897ffe995bbcef480, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:54:27,375 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/67f9d0e4feaf4b538606a6c019d1ae07 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/67f9d0e4feaf4b538606a6c019d1ae07 2023-10-27 23:54:27,378 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/67f9d0e4feaf4b538606a6c019d1ae07, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:54:27,379 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 8f308088886703aa0a6843bebc9038fd in 30ms, sequenceid=71, compaction requested=true 2023-10-27 23:54:27,379 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8f308088886703aa0a6843bebc9038fd: 2023-10-27 23:54:27,379 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:27,379 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:27,379 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:27,379 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,379 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-10-27 23:54:27,379 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-10-27 23:54:27,388 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,388 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:27,389 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,389 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:27,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-10-27 23:54:27,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ea598eb[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:27,390 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-10-27 23:54:27,391 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,399 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,399 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,399 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,400 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:27,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,400 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,400 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,400 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:27,400 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:27,400 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:27,400 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:27,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,400 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,400 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-10-27 23:54:27,400 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-10-27 23:54:27,400 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:27,408 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:27,408 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:27,408 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:27,408 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:27,408 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:27,408 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,408 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:27,416 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-10-27 23:54:27,416 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:27,416 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-10-27 23:54:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:54:27,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-10-27 23:54:27,419 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-10-27 23:54:27,419 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:27,419 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:27,419 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-10-27 23:54:27,419 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:27,430 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,430 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,430 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,430 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:27,430 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:27,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,430 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:27,430 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:27,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:27,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:27,430 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:27,431 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:27,435 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-10-27 23:54:27,436 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,437 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,437 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,446 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,446 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:27,446 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:27,446 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:27,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:27,447 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:27,446 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:27,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:27,446 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:27,446 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:27,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-10-27 23:54:27,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:27,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:27,447 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:27,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:27,447 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:27,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:27,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:27,447 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:27,447 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:27,447 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,448 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-10-27 23:54:27,448 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,448 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,448 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:27,448 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:27,448 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-10-27 23:54:27,448 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-10-27 23:54:27,548 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:27,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:27,555 INFO [Listener at localhost/43373] assignment.SplitTableRegionProcedure(206): testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. splitKey isn't explicitly specified, will try to find a best split key from RS jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,556 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.StoreUtils(129): cannot split hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/f6600f6e12724573ac25d59aaa99f951 because midkey is the same as first or last row 2023-10-27 23:54:27,556 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.StoreUtils(129): cannot split hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/17f319a4fe88455a9585ab4df0ba18b5 because midkey is the same as first or last row 2023-10-27 23:54:27,556 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.HRegion(2745): Flushing 8f308088886703aa0a6843bebc9038fd 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-10-27 23:54:27,564 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/e6472601172346828a7455c58fe4f9a1 2023-10-27 23:54:27,575 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/f8077f33bdfd40c993e8c7d20c389806 2023-10-27 23:54:27,579 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf1/e6472601172346828a7455c58fe4f9a1 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/e6472601172346828a7455c58fe4f9a1 2023-10-27 23:54:27,583 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/e6472601172346828a7455c58fe4f9a1, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:54:27,584 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/.tmp/cf2/f8077f33bdfd40c993e8c7d20c389806 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/f8077f33bdfd40c993e8c7d20c389806 2023-10-27 23:54:27,588 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/f8077f33bdfd40c993e8c7d20c389806, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:54:27,589 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 8f308088886703aa0a6843bebc9038fd in 33ms, sequenceid=82, compaction requested=true 2023-10-27 23:54:27,590 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.HRegion(2446): Flush status journal for 8f308088886703aa0a6843bebc9038fd: 2023-10-27 23:54:27,590 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.StoreUtils(129): cannot split hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/f6600f6e12724573ac25d59aaa99f951 because midkey is the same as first or last row 2023-10-27 23:54:27,590 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39059] regionserver.StoreUtils(129): cannot split hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/17f319a4fe88455a9585ab4df0ba18b5 because midkey is the same as first or last row 2023-10-27 23:54:27,590 DEBUG [Listener at localhost/43373] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,591 DEBUG [Listener at localhost/43373] assignment.TestSplitTableRegionProcedure(438): Expected Split procedure construction failure: Region not splittable because bestSplitPoint = null, maybe table is too small for auto split. For force split, try specifying split row 2023-10-27 23:54:27,591 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:54:27,592 INFO [Listener at localhost/43373] client.HBaseAdmin$15(890): Started disable of testInvalidSplitKey 2023-10-27 23:54:27,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable testInvalidSplitKey 2023-10-27 23:54:27,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:27,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=105 2023-10-27 23:54:27,596 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450867595"}]},"ts":"1698450867595"} 2023-10-27 23:54:27,596 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=DISABLING in hbase:meta 2023-10-27 23:54:27,605 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testInvalidSplitKey to state=DISABLING 2023-10-27 23:54:27,606 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testInvalidSplitKey, region=8f308088886703aa0a6843bebc9038fd, UNASSIGN}] 2023-10-27 23:54:27,606 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testInvalidSplitKey, region=8f308088886703aa0a6843bebc9038fd, UNASSIGN 2023-10-27 23:54:27,607 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=8f308088886703aa0a6843bebc9038fd, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,607 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1698450867607"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450867607"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450867607"}]},"ts":"1698450867607"} 2023-10-27 23:54:27,608 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=107, ppid=106, state=RUNNABLE; CloseRegionProcedure 8f308088886703aa0a6843bebc9038fd, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:54:27,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=105 2023-10-27 23:54:27,760 DEBUG [RSProcedureDispatcher-pool-3] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:27,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:27,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 8f308088886703aa0a6843bebc9038fd, disabling compactions & flushes 2023-10-27 23:54:27,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:27,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:27,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. after waiting 0 ms 2023-10-27 23:54:27,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:27,764 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-10-27 23:54:27,765 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:27,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd. 2023-10-27 23:54:27,765 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 8f308088886703aa0a6843bebc9038fd: 2023-10-27 23:54:27,766 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:27,766 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=8f308088886703aa0a6843bebc9038fd, regionState=CLOSED 2023-10-27 23:54:27,767 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1698450867766"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450867766"}]},"ts":"1698450867766"} 2023-10-27 23:54:27,769 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=107, resume processing ppid=106 2023-10-27 23:54:27,769 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=107, ppid=106, state=SUCCESS; CloseRegionProcedure 8f308088886703aa0a6843bebc9038fd, server=jenkins-hbase16.apache.org,39059,1698450767147 in 160 msec 2023-10-27 23:54:27,770 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=106, resume processing ppid=105 2023-10-27 23:54:27,770 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testInvalidSplitKey, region=8f308088886703aa0a6843bebc9038fd, UNASSIGN in 163 msec 2023-10-27 23:54:27,771 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450867770"}]},"ts":"1698450867770"} 2023-10-27 23:54:27,772 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=DISABLED in hbase:meta 2023-10-27 23:54:27,780 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testInvalidSplitKey to state=DISABLED 2023-10-27 23:54:27,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey in 187 msec 2023-10-27 23:54:27,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=105 2023-10-27 23:54:27,898 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testInvalidSplitKey, procId: 105 completed 2023-10-27 23:54:27,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete testInvalidSplitKey 2023-10-27 23:54:27,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=108, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:27,901 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=108, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:27,902 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=108, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:27,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=108 2023-10-27 23:54:27,904 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:27,906 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/recovered.edits] 2023-10-27 23:54:27,910 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/42750e8b000145bc9fc23356929d3fbd to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/42750e8b000145bc9fc23356929d3fbd 2023-10-27 23:54:27,911 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/6cca79f962be4277a64d88f342e41d36 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/6cca79f962be4277a64d88f342e41d36 2023-10-27 23:54:27,912 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/9c9c41281b6e407897ffe995bbcef480 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/9c9c41281b6e407897ffe995bbcef480 2023-10-27 23:54:27,913 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/d98880e3ea30496d94be2a0407539c8c to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/d98880e3ea30496d94be2a0407539c8c 2023-10-27 23:54:27,913 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/e6472601172346828a7455c58fe4f9a1 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/e6472601172346828a7455c58fe4f9a1 2023-10-27 23:54:27,914 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/f6600f6e12724573ac25d59aaa99f951 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/f6600f6e12724573ac25d59aaa99f951 2023-10-27 23:54:27,916 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/fc7761a8754748359e67682a0f9f5891 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf1/fc7761a8754748359e67682a0f9f5891 2023-10-27 23:54:27,918 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/17f319a4fe88455a9585ab4df0ba18b5 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/17f319a4fe88455a9585ab4df0ba18b5 2023-10-27 23:54:27,919 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/3925004ad5814455993415b083261b8a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/3925004ad5814455993415b083261b8a 2023-10-27 23:54:27,919 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/4103850a0cc84fe5ae825944ee6ace84 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/4103850a0cc84fe5ae825944ee6ace84 2023-10-27 23:54:27,920 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/5dbc00e8b1084e1cbc0099c25976383f to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/5dbc00e8b1084e1cbc0099c25976383f 2023-10-27 23:54:27,921 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/67f9d0e4feaf4b538606a6c019d1ae07 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/67f9d0e4feaf4b538606a6c019d1ae07 2023-10-27 23:54:27,922 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/9da1dbbb713f45a3be98ce38907c3526 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/9da1dbbb713f45a3be98ce38907c3526 2023-10-27 23:54:27,923 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/f8077f33bdfd40c993e8c7d20c389806 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/cf2/f8077f33bdfd40c993e8c7d20c389806 2023-10-27 23:54:27,924 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/recovered.edits/85.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd/recovered.edits/85.seqid 2023-10-27 23:54:27,925 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testInvalidSplitKey/8f308088886703aa0a6843bebc9038fd 2023-10-27 23:54:27,925 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testInvalidSplitKey regions 2023-10-27 23:54:27,926 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=108, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:27,927 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testInvalidSplitKey from hbase:meta 2023-10-27 23:54:27,931 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testInvalidSplitKey' descriptor. 2023-10-27 23:54:27,932 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=108, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:27,932 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testInvalidSplitKey' from region states. 2023-10-27 23:54:27,932 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450867932"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:27,933 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-10-27 23:54:27,934 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8f308088886703aa0a6843bebc9038fd, NAME => 'testInvalidSplitKey,,1698450865342.8f308088886703aa0a6843bebc9038fd.', STARTKEY => '', ENDKEY => ''}] 2023-10-27 23:54:27,934 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testInvalidSplitKey' as deleted. 2023-10-27 23:54:27,934 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1698450867934"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:27,935 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testInvalidSplitKey state from META 2023-10-27 23:54:27,942 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=108, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:27,943 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey in 43 msec 2023-10-27 23:54:28,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=108 2023-10-27 23:54:28,003 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testInvalidSplitKey, procId: 108 completed 2023-10-27 23:54:28,015 INFO [Listener at localhost/43373] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testInvalidSplitKey Thread=370 (was 351) - Thread LEAK? -, OpenFileDescriptor=742 (was 658) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=225 (was 225), ProcessCount=171 (was 171), AvailableMemoryMB=5466 (was 5483) 2023-10-27 23:54:28,027 INFO [Listener at localhost/43373] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRecoveryAndDoubleExecution Thread=370, OpenFileDescriptor=742, MaxFileDescriptor=60000, SystemLoadAverage=225, ProcessCount=171, AvailableMemoryMB=5466 2023-10-27 23:54:28,028 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:54:28,038 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-10-27 23:54:28,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(492): Client=jenkins//188.40.62.62 set balanceSwitch=false 2023-10-27 23:54:28,040 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=109, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:28,041 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_PRE_OPERATION 2023-10-27 23:54:28,042 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-10-27 23:54:28,043 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:28,044 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55 empty. 2023-10-27 23:54:28,044 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:28,044 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRecoveryAndDoubleExecution regions 2023-10-27 23:54:28,052 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/.tabledesc/.tableinfo.0000000001 2023-10-27 23:54:28,052 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(7675): creating {ENCODED => 24b4632255f0f3b88a079696a3d55a55, NAME => 'testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRecoveryAndDoubleExecution', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp 2023-10-27 23:54:28,059 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(866): Instantiated testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:28,059 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1604): Closing 24b4632255f0f3b88a079696a3d55a55, disabling compactions & flushes 2023-10-27 23:54:28,059 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1626): Closing region testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:28,059 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:28,059 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1714): Acquired close lock on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. after waiting 0 ms 2023-10-27 23:54:28,059 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1724): Updates disabled for region testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:28,059 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1838): Closed testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:28,059 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1558): Region close journal for 24b4632255f0f3b88a079696a3d55a55: 2023-10-27 23:54:28,061 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_ADD_TO_META 2023-10-27 23:54:28,062 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450868062"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450868062"}]},"ts":"1698450868062"} 2023-10-27 23:54:28,063 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-10-27 23:54:28,064 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-10-27 23:54:28,064 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450868064"}]},"ts":"1698450868064"} 2023-10-27 23:54:28,065 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=ENABLING in hbase:meta 2023-10-27 23:54:28,083 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-10-27 23:54:28,084 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-27 23:54:28,084 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-10-27 23:54:28,084 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-10-27 23:54:28,084 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-10-27 23:54:28,084 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, ASSIGN}] 2023-10-27 23:54:28,086 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, ASSIGN 2023-10-27 23:54:28,086 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,43641,1698450767195; forceNewPlan=false, retain=false 2023-10-27 23:54:28,237 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-10-27 23:54:28,238 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=24b4632255f0f3b88a079696a3d55a55, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,238 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450868238"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450868238"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450868238"}]},"ts":"1698450868238"} 2023-10-27 23:54:28,239 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=111, ppid=110, state=RUNNABLE; OpenRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:54:28,390 DEBUG [RSProcedureDispatcher-pool-4] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,392 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:28,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 24b4632255f0f3b88a079696a3d55a55, NAME => 'testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:54:28,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:54:28,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:28,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:28,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:28,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:28,394 INFO [StoreOpener-24b4632255f0f3b88a079696a3d55a55-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:28,396 DEBUG [StoreOpener-24b4632255f0f3b88a079696a3d55a55-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1 2023-10-27 23:54:28,396 DEBUG [StoreOpener-24b4632255f0f3b88a079696a3d55a55-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1 2023-10-27 23:54:28,396 INFO [StoreOpener-24b4632255f0f3b88a079696a3d55a55-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 24b4632255f0f3b88a079696a3d55a55 columnFamilyName cf1 2023-10-27 23:54:28,397 INFO [StoreOpener-24b4632255f0f3b88a079696a3d55a55-1] regionserver.HStore(310): Store=24b4632255f0f3b88a079696a3d55a55/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:28,397 INFO [StoreOpener-24b4632255f0f3b88a079696a3d55a55-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:28,398 DEBUG [StoreOpener-24b4632255f0f3b88a079696a3d55a55-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2 2023-10-27 23:54:28,398 DEBUG [StoreOpener-24b4632255f0f3b88a079696a3d55a55-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2 2023-10-27 23:54:28,398 INFO [StoreOpener-24b4632255f0f3b88a079696a3d55a55-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 24b4632255f0f3b88a079696a3d55a55 columnFamilyName cf2 2023-10-27 23:54:28,399 INFO [StoreOpener-24b4632255f0f3b88a079696a3d55a55-1] regionserver.HStore(310): Store=24b4632255f0f3b88a079696a3d55a55/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:28,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:28,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:28,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:54:28,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:28,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-10-27 23:54:28,404 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 24b4632255f0f3b88a079696a3d55a55; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64823932, jitterRate=-0.03404814004898071}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:54:28,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 24b4632255f0f3b88a079696a3d55a55: 2023-10-27 23:54:28,405 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55., pid=111, masterSystemTime=1698450868390 2023-10-27 23:54:28,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:28,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:28,407 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=24b4632255f0f3b88a079696a3d55a55, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,407 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450868407"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450868407"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450868407"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450868407"}]},"ts":"1698450868407"} 2023-10-27 23:54:28,409 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=111, resume processing ppid=110 2023-10-27 23:54:28,409 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=111, ppid=110, state=SUCCESS; OpenRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 in 169 msec 2023-10-27 23:54:28,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=110, resume processing ppid=109 2023-10-27 23:54:28,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=110, ppid=109, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, ASSIGN in 325 msec 2023-10-27 23:54:28,410 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-10-27 23:54:28,411 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450868411"}]},"ts":"1698450868411"} 2023-10-27 23:54:28,411 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=ENABLED in hbase:meta 2023-10-27 23:54:28,422 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_POST_OPERATION 2023-10-27 23:54:28,423 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution in 383 msec 2023-10-27 23:54:28,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:28,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-10-27 23:54:28,549 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-10-27 23:54:28,549 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:28,549 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-10-27 23:54:28,549 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-10-27 23:54:28,549 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,549 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,558 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,558 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:28,558 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:28,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:28,558 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:28,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:28,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:28,558 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,558 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:28,559 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:28,559 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:28,559 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:28,559 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:28,560 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:28,559 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:28,560 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:28,560 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:28,560 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:28,560 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:28,560 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:28,560 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:28,560 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:28,560 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:28,560 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:28,560 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. started... 2023-10-27 23:54:28,561 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 24b4632255f0f3b88a079696a3d55a55 2/2 column families, dataSize=120 B heapSize=928 B 2023-10-27 23:54:28,566 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,569 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/ab6cf8f8528e4010ac620294cd4a93a7 2023-10-27 23:54:28,574 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,574 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,574 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:28,574 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,575 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:28,580 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/fb908c093a2e4a2495e37ed55e98e741 2023-10-27 23:54:28,584 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/ab6cf8f8528e4010ac620294cd4a93a7 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7 2023-10-27 23:54:28,588 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:54:28,588 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/fb908c093a2e4a2495e37ed55e98e741 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741 2023-10-27 23:54:28,592 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:54:28,592 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 24b4632255f0f3b88a079696a3d55a55 in 31ms, sequenceid=6, compaction requested=false 2023-10-27 23:54:28,592 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRecoveryAndDoubleExecution' 2023-10-27 23:54:28,593 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 24b4632255f0f3b88a079696a3d55a55: 2023-10-27 23:54:28,593 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:28,593 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:28,593 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:28,593 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,593 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:28,593 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:28,599 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,599 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:28,600 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,600 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:28,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-10-27 23:54:28,602 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-10-27 23:54:28,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3400ffbb[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:28,602 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,608 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,608 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,608 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,608 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,608 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:28,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,608 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:28,608 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:28,608 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,608 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:28,608 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:28,608 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:28,608 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:28,608 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:28,609 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:28,616 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:28,616 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:28,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,616 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:28,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:28,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:28,624 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:28,624 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:28,624 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:28,633 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:28,633 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:28,633 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:54:28,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-10-27 23:54:28,636 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-10-27 23:54:28,636 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:28,636 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:28,636 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-10-27 23:54:28,636 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:28,646 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,646 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,646 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:28,646 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,646 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,646 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,647 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,646 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:28,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:28,647 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,647 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,648 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:28,649 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:28,649 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,696 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:28,696 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:28,696 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:28,696 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:28,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-10-27 23:54:28,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:28,696 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,697 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,697 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:28,697 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:28,697 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:28,696 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:28,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:28,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:28,696 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:28,697 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:28,697 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:28,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:28,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:28,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:28,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,698 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,798 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:28,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:28,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:28,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:28,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-10-27 23:54:28,810 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-10-27 23:54:28,810 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:28,810 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-10-27 23:54:28,810 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-10-27 23:54:28,811 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,811 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,816 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:28,816 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,816 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:28,816 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:28,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:28,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:28,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:28,817 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,817 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:28,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:28,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:28,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:28,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:28,818 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:28,818 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:28,818 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. started... 2023-10-27 23:54:28,819 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 24b4632255f0f3b88a079696a3d55a55 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:28,826 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/5da2fdb2de0741f5a19326bd3a22c207 2023-10-27 23:54:28,835 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/19cd9f6c45444674ad42782a1dcfbeaf 2023-10-27 23:54:28,837 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,839 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/5da2fdb2de0741f5a19326bd3a22c207 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207 2023-10-27 23:54:28,842 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:54:28,843 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/19cd9f6c45444674ad42782a1dcfbeaf as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf 2023-10-27 23:54:28,847 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:54:28,848 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 24b4632255f0f3b88a079696a3d55a55 in 30ms, sequenceid=19, compaction requested=false 2023-10-27 23:54:28,848 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 24b4632255f0f3b88a079696a3d55a55: 2023-10-27 23:54:28,848 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:28,848 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:28,848 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:28,848 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,848 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:28,848 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:28,849 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,849 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,849 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:28,858 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,858 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,858 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,858 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:28,858 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,858 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:28,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-10-27 23:54:28,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a9a9a32[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:28,860 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-10-27 23:54:28,860 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,866 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,866 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,866 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,866 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:28,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,866 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:28,866 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:28,866 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,866 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:28,867 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:28,866 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:28,866 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:28,866 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:28,867 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:28,867 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:28,874 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:28,874 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:28,874 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,874 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,874 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:28,874 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:28,874 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:28,883 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:28,883 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:28,883 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:54:28,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-10-27 23:54:28,886 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-10-27 23:54:28,886 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:28,886 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:28,886 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-10-27 23:54:28,886 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:28,896 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,896 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,896 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,896 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,897 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:28,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,896 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:28,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,897 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,896 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:28,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:28,897 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:28,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,897 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,899 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,899 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,899 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:28,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,908 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,908 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:28,908 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:28,908 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:28,908 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:28,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:28,908 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:28,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-10-27 23:54:28,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:28,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:28,908 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:28,908 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:28,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:28,908 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:28,908 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:28,909 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:28,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:28,908 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,909 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,909 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,909 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:28,909 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,909 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:28,909 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:28,909 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:28,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:28,909 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,009 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:29,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:29,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:29,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:29,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-10-27 23:54:29,022 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-10-27 23:54:29,023 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,023 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-10-27 23:54:29,023 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-10-27 23:54:29,023 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,023 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,033 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,033 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,033 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,033 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,033 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,033 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:29,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:29,034 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:29,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,035 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:29,035 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,034 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,035 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:29,035 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:29,035 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,035 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:29,035 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:29,035 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:29,035 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:29,035 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:29,035 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:29,035 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:29,035 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:29,035 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:29,035 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. started... 2023-10-27 23:54:29,035 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 24b4632255f0f3b88a079696a3d55a55 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:29,041 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,041 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,041 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,041 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,041 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:29,041 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:29,043 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/51f737396e99468cbd1e87a1c8088e6f 2023-10-27 23:54:29,056 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/df0c477557384f79a606523bb4f58f8a 2023-10-27 23:54:29,059 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/51f737396e99468cbd1e87a1c8088e6f as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f 2023-10-27 23:54:29,063 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:54:29,064 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/df0c477557384f79a606523bb4f58f8a as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a 2023-10-27 23:54:29,067 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:54:29,068 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 24b4632255f0f3b88a079696a3d55a55 in 33ms, sequenceid=32, compaction requested=true 2023-10-27 23:54:29,068 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 24b4632255f0f3b88a079696a3d55a55: 2023-10-27 23:54:29,068 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:29,068 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:29,068 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:29,068 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,068 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:29,068 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:29,074 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,074 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:29,075 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,075 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:29,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-10-27 23:54:29,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b6ef80b[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:29,076 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-10-27 23:54:29,076 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,083 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,083 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,083 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,083 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:29,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,083 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,083 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:29,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,083 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:29,083 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,083 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:29,083 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:29,083 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:29,083 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:29,083 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:29,083 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:29,091 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,091 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:29,091 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,091 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:29,099 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:29,099 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,099 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:29,122 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:29,122 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,122 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:54:29,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-10-27 23:54:29,125 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-10-27 23:54:29,125 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:29,125 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:29,125 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,125 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:29,138 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,138 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,138 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,138 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:29,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,138 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,138 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,138 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,139 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,139 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,139 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:29,140 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,141 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,141 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,149 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,149 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,149 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,149 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,150 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,149 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-10-27 23:54:29,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:29,150 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,150 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,150 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,150 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:29,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,150 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,150 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:29,150 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,150 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,150 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,150 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,150 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,151 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,151 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,151 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,250 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:29,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:29,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:29,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:29,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-10-27 23:54:29,265 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-10-27 23:54:29,265 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,265 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-10-27 23:54:29,265 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-10-27 23:54:29,266 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,266 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,274 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,274 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,274 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,274 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,275 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,275 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:29,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:29,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:29,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:29,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,276 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:29,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:29,276 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:29,276 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:29,276 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:29,276 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,276 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:29,276 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:29,276 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:29,276 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:29,276 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:29,276 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:29,276 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,276 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:29,276 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:29,276 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. started... 2023-10-27 23:54:29,277 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:29,277 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:29,277 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 24b4632255f0f3b88a079696a3d55a55 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:29,282 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,284 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/a0f5d933fe754195aed6fca431e18ce8 2023-10-27 23:54:29,291 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,291 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,291 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:29,291 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,291 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:29,294 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/94921a45fbff4ff28109056240e59ec4 2023-10-27 23:54:29,298 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/a0f5d933fe754195aed6fca431e18ce8 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8 2023-10-27 23:54:29,301 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:54:29,301 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/94921a45fbff4ff28109056240e59ec4 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4 2023-10-27 23:54:29,305 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:54:29,305 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 24b4632255f0f3b88a079696a3d55a55 in 28ms, sequenceid=45, compaction requested=true 2023-10-27 23:54:29,306 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 24b4632255f0f3b88a079696a3d55a55: 2023-10-27 23:54:29,306 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:29,306 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:29,306 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:29,306 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,306 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:29,306 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:29,316 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,316 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:29,317 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:29,317 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-10-27 23:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69ce4f4d[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:29,318 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-10-27 23:54:29,318 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,324 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,324 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,324 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,325 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,325 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,325 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:29,325 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:29,325 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:29,325 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:29,325 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:29,325 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:29,325 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:29,325 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:29,325 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:29,333 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:29,333 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,333 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:29,341 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:29,341 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,341 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:29,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:29,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:29,341 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,341 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,341 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:29,341 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:54:29,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-10-27 23:54:29,344 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-10-27 23:54:29,344 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:29,344 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:29,344 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,344 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:29,354 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,354 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,354 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:29,354 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,355 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,355 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,355 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,355 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,356 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,356 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,357 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,357 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,357 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,366 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-10-27 23:54:29,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,366 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,367 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,367 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:29,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,367 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:29,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,367 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,367 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,367 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,367 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,368 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,368 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,368 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,467 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:29,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:29,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:29,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:29,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-10-27 23:54:29,482 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-10-27 23:54:29,482 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,483 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-10-27 23:54:29,483 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-10-27 23:54:29,483 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,483 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,491 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,491 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,491 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,491 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,491 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,491 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:29,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:29,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:29,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:29,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:29,493 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:29,493 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. started... 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:29,493 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 24b4632255f0f3b88a079696a3d55a55 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:29,493 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:29,505 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/46a9bf3709a942f5867e091b8f085cbb 2023-10-27 23:54:29,516 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/a78b6261d9f0484790e834172b9147a4 2023-10-27 23:54:29,520 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/46a9bf3709a942f5867e091b8f085cbb as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb 2023-10-27 23:54:29,523 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,524 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:54:29,525 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/a78b6261d9f0484790e834172b9147a4 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4 2023-10-27 23:54:29,529 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:54:29,530 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 24b4632255f0f3b88a079696a3d55a55 in 37ms, sequenceid=58, compaction requested=true 2023-10-27 23:54:29,530 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 24b4632255f0f3b88a079696a3d55a55: 2023-10-27 23:54:29,530 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:29,530 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:29,530 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:29,530 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,530 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:29,530 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:29,533 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,533 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,533 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:29,533 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,533 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:29,541 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,541 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:29,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:29,541 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,542 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:29,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-10-27 23:54:29,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a0098e8[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:29,544 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-10-27 23:54:29,544 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,549 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,549 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,549 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,550 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,550 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,550 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:29,550 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:29,550 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,550 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:29,550 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:29,550 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:29,550 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:29,550 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:29,550 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:29,558 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:29,558 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,558 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:29,558 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:29,566 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:29,566 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,566 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:29,566 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:29,566 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,566 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-10-27 23:54:29,569 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-10-27 23:54:29,569 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:29,569 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:29,569 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,569 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:29,580 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,580 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,580 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,580 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,580 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,580 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,580 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,580 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,581 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,581 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:29,582 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,582 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,583 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,591 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,591 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,591 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,591 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,591 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,591 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,591 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,591 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,591 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-10-27 23:54:29,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:29,591 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,592 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,592 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,592 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:29,592 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,592 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:29,592 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,592 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,592 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,592 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,592 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,692 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:29,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:29,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:29,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:29,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-10-27 23:54:29,707 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-10-27 23:54:29,707 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,707 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-10-27 23:54:29,707 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-10-27 23:54:29,707 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,707 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,716 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,716 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,716 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,716 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,717 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,717 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:29,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:29,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-10-27 23:54:29,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:29,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:29,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-10-27 23:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:29,718 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:29,718 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:29,719 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:29,719 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. started... 2023-10-27 23:54:29,719 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 24b4632255f0f3b88a079696a3d55a55 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:29,724 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,727 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/1b440e72bfc24900bf116a81845193a4 2023-10-27 23:54:29,732 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,732 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,733 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:29,733 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,733 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:29,741 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/e2137e2a2fbe4f239761d6c3b26491c3 2023-10-27 23:54:29,745 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/1b440e72bfc24900bf116a81845193a4 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4 2023-10-27 23:54:29,749 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:54:29,749 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/e2137e2a2fbe4f239761d6c3b26491c3 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3 2023-10-27 23:54:29,753 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:54:29,754 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 24b4632255f0f3b88a079696a3d55a55 in 35ms, sequenceid=71, compaction requested=true 2023-10-27 23:54:29,754 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 24b4632255f0f3b88a079696a3d55a55: 2023-10-27 23:54:29,754 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:29,754 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:29,754 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:29,754 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,754 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-10-27 23:54:29,754 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-10-27 23:54:29,763 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,763 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:29,763 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,764 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:29,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-10-27 23:54:29,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34a5d290[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:29,766 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-10-27 23:54:29,766 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,774 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,774 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,774 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,775 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:29,775 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:29,775 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,775 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:29,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,775 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,775 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:29,775 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:29,775 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-10-27 23:54:29,775 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-10-27 23:54:29,775 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:29,775 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:29,783 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:29,783 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,783 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:29,807 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:29,807 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,807 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:29,807 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,807 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-10-27 23:54:29,807 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,808 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-10-27 23:54:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:54:29,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-10-27 23:54:29,811 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-10-27 23:54:29,811 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:29,811 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:29,811 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-10-27 23:54:29,811 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:29,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,825 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,825 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,825 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:29,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,826 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,826 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,827 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:29,827 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-10-27 23:54:29,828 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:29,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,838 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:29,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-10-27 23:54:29,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:29,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:29,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:29,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,839 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:29,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:29,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,839 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:29,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:29,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:29,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:29,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-10-27 23:54:29,939 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:29,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:30,198 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-10-27 23:54:30,198 DEBUG [Listener at localhost/43373] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:30,199 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,200 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,200 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,200 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,201 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:54:30,201 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,201 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:54:30,201 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:54:30,449 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(454): Restart 0 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,450 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:54:30,450 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:54:30,450 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:54:30,450 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-9,5,PEWorkerGroup] Thread[HFileArchiver-10,5,PEWorkerGroup] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] 2023-10-27 23:54:30,450 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:54:30,450 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:54:30,450 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:54:30,450 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:54:30,451 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:54:30,453 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-10-27 23:54:30,453 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:54:30,454 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:54:30,454 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:54:30,454 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:54:30,454 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:30,454 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:54:30,455 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:30,455 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:30,455 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:30,455 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:54:30,455 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:30,456 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:30,456 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:30,456 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:54:30,456 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:54:30,457 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:30,457 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:30,457 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:30,457 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:54:30,457 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:30,457 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:30,458 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:30,458 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:54:30,458 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:30,458 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:30,459 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:30,459 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:30,459 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:30,460 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:30,460 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:30,460 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:30,460 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:30,460 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:30,460 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:30,461 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,461 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:54:30,461 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,461 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,461 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,461 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 9 msec 2023-10-27 23:54:30,462 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:54:30,463 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:54:30,463 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450900463 2023-10-27 23:54:30,464 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:54:30,464 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,466 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:54:30,467 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=24b4632255f0f3b88a079696a3d55a55, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:54:30,467 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:54:30,467 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450930467 2023-10-27 23:54:30,467 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450990467 2023-10-27 23:54:30,467 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:54:30,467 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:54:30,470 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:54:30,470 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,470 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:54:30,470 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:54:30,626 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-27 23:54:30,717 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(454): Restart 1 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,718 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:54:30,718 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:54:30,718 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:54:30,718 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:54:30,718 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:54:30,718 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:54:30,718 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:54:30,718 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:54:30,719 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:54:30,720 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-10-27 23:54:30,720 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:54:30,720 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:54:30,720 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:54:30,721 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:54:30,721 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:30,721 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:54:30,721 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:30,721 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:30,721 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:30,721 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:54:30,721 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:30,721 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:30,722 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:30,722 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:54:30,722 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:54:30,722 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:30,722 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:30,722 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:30,722 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:54:30,722 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:30,723 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:30,723 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:30,723 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:54:30,723 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:30,723 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:30,723 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:30,723 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:30,723 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:30,724 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:30,724 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:30,724 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:30,724 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:30,724 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:30,724 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:30,724 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,724 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:54:30,724 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,724 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,725 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,725 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 5 msec 2023-10-27 23:54:30,725 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:54:30,726 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:54:30,726 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450900726 2023-10-27 23:54:30,728 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,728 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:54:30,730 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:54:30,730 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=24b4632255f0f3b88a079696a3d55a55, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:54:30,730 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:54:30,730 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450930730 2023-10-27 23:54:30,730 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450990730 2023-10-27 23:54:30,730 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 2 msec 2023-10-27 23:54:30,731 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:54:30,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN}] 2023-10-27 23:54:30,733 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:54:30,733 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:54:30,733 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:54:30,981 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(454): Restart 2 exec state=pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,981 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:54:30,981 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:54:30,981 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:54:30,981 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:54:30,981 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:54:30,981 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:54:30,981 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:54:30,981 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:54:30,982 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:54:30,983 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-10-27 23:54:30,983 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:54:30,984 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:54:30,984 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:54:30,984 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:54:30,984 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:30,984 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:54:30,984 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:30,984 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:30,985 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:30,985 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:54:30,985 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:30,985 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:30,985 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:30,985 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:54:30,985 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:54:30,985 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:30,985 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:30,986 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:30,986 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:54:30,986 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:30,986 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:30,986 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:30,986 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:54:30,986 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:30,986 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:30,987 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:30,987 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:30,987 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:30,987 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:30,987 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:30,987 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:30,988 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:30,988 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:30,988 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:30,988 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,988 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:54:30,988 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,988 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,988 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,988 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 6 msec 2023-10-27 23:54:30,989 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:54:30,989 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:54:30,990 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450900990 2023-10-27 23:54:30,991 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:54:30,991 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:30,993 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:54:30,993 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=24b4632255f0f3b88a079696a3d55a55, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:54:30,993 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:54:30,993 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450930993 2023-10-27 23:54:30,993 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450990993 2023-10-27 23:54:30,994 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 2 msec 2023-10-27 23:54:30,994 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN}] 2023-10-27 23:54:30,994 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:54:30,996 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN 2023-10-27 23:54:30,997 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=113 updating hbase:meta row=24b4632255f0f3b88a079696a3d55a55, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:30,997 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450870996"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450870996"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450870996"}]},"ts":"1698450870996"} 2023-10-27 23:54:30,998 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=114, ppid=113, state=RUNNABLE; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:54:31,149 DEBUG [RSProcedureDispatcher-pool-5] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:31,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 24b4632255f0f3b88a079696a3d55a55, disabling compactions & flushes 2023-10-27 23:54:31,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:31,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:31,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. after waiting 0 ms 2023-10-27 23:54:31,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:31,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 24b4632255f0f3b88a079696a3d55a55 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-10-27 23:54:31,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/f62fe57988ed466b844ebb088bdd1fdf 2023-10-27 23:54:31,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/eca3177bff2f41caa13763f4b034c8e2 2023-10-27 23:54:31,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf1/f62fe57988ed466b844ebb088bdd1fdf as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf 2023-10-27 23:54:31,175 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:54:31,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/.tmp/cf2/eca3177bff2f41caa13763f4b034c8e2 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2 2023-10-27 23:54:31,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:54:31,180 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 24b4632255f0f3b88a079696a3d55a55 in 30ms, sequenceid=82, compaction requested=true 2023-10-27 23:54:31,184 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-10-27 23:54:31,184 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:31,185 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. 2023-10-27 23:54:31,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 24b4632255f0f3b88a079696a3d55a55: 2023-10-27 23:54:31,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,187 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=113 updating hbase:meta row=24b4632255f0f3b88a079696a3d55a55, regionState=CLOSED 2023-10-27 23:54:31,187 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450871187"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450871187"}]},"ts":"1698450871187"} 2023-10-27 23:54:31,190 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=114, resume processing ppid=113 2023-10-27 23:54:31,190 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 in 191 msec 2023-10-27 23:54:31,191 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=113, resume processing ppid=112 2023-10-27 23:54:31,191 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN in 196 msec 2023-10-27 23:54:31,193 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:54:31,193 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,193 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:54:31,193 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:54:31,244 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(454): Restart 3 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,244 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:54:31,244 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:54:31,244 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:54:31,244 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:54:31,244 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:54:31,244 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:54:31,244 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:54:31,244 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:54:31,245 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:54:31,246 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-10-27 23:54:31,246 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:54:31,246 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:54:31,247 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:54:31,247 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:54:31,247 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:54:31,247 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:54:31,247 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:31,247 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:54:31,247 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:31,248 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:31,248 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:31,248 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:54:31,248 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:31,248 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:31,248 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:31,248 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:54:31,249 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:54:31,249 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:31,249 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:31,249 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:31,249 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:54:31,249 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:31,249 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:31,249 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:31,249 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:54:31,250 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:31,250 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:31,250 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:31,250 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:31,250 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:31,250 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:31,250 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:31,250 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:31,251 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:31,251 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:31,251 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:31,251 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,251 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN 2023-10-27 23:54:31,251 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:31,251 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:54:31,251 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,251 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,251 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,251 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:31,252 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:31,252 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 6 msec 2023-10-27 23:54:31,252 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:54:31,253 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:54:31,253 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450901253 2023-10-27 23:54:31,253 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:54:31,254 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,257 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:54:31,257 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=24b4632255f0f3b88a079696a3d55a55, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:54:31,257 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:54:31,257 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450931257 2023-10-27 23:54:31,257 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450991257 2023-10-27 23:54:31,257 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 4 msec 2023-10-27 23:54:31,260 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:54:31,270 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=112 splitting 14 storefiles, region=24b4632255f0f3b88a079696a3d55a55, threads=14 2023-10-27 23:54:31,270 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,270 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,270 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,270 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,270 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,271 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,274 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,274 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,274 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,274 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,275 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,275 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,279 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,279 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,430 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,455 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,455 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,472 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,473 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,479 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,479 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,480 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,483 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,483 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,483 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,493 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,495 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,496 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,496 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=112 split storefiles for region 24b4632255f0f3b88a079696a3d55a55 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-10-27 23:54:31,522 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:54:31,522 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,522 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:54:31,522 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:54:31,758 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(454): Restart 4 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,758 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:54:31,758 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:54:31,758 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:54:31,758 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:54:31,758 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:54:31,758 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:54:31,758 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:54:31,758 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:54:31,759 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:54:31,760 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-10-27 23:54:31,760 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:54:31,760 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:54:31,760 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:54:31,761 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:54:31,761 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:54:31,761 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:54:31,761 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:31,761 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:54:31,761 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:31,762 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:31,762 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:31,762 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:54:31,762 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:31,762 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:31,762 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:31,762 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:54:31,763 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:54:31,763 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:31,763 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:31,763 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:31,763 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:54:31,763 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:31,763 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:31,763 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:31,764 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:54:31,764 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:31,764 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:31,764 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:31,764 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:31,764 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:31,764 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:31,764 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:31,765 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:31,765 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:31,765 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:31,765 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:31,765 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,765 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN 2023-10-27 23:54:31,765 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:31,765 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:54:31,765 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,765 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,766 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,766 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:31,766 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:31,766 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 6 msec 2023-10-27 23:54:31,767 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:54:31,767 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:54:31,767 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450901767 2023-10-27 23:54:31,769 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:31,769 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:54:31,772 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:54:31,772 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=24b4632255f0f3b88a079696a3d55a55, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:54:31,772 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:54:31,772 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450931772 2023-10-27 23:54:31,772 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450991772 2023-10-27 23:54:31,772 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:54:31,780 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=112 splitting 14 storefiles, region=24b4632255f0f3b88a079696a3d55a55, threads=14 2023-10-27 23:54:31,780 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,783 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,787 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,787 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,791 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,793 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,794 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,794 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,794 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,794 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,794 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,795 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,795 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,799 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,937 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,944 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,945 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,955 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,967 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,967 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,970 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,971 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,977 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,977 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,978 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,979 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,979 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,980 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8 for region: 24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:31,980 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=112 split storefiles for region 24b4632255f0f3b88a079696a3d55a55 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-10-27 23:54:32,008 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:54:32,013 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-10-27 23:54:32,015 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-10-27 23:54:32,015 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:54:32,015 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:54:32,015 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:54:32,022 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(454): Restart 5 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,022 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:54:32,023 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:54:32,023 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:54:32,023 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:54:32,023 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:54:32,023 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:54:32,023 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:54:32,023 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:54:32,024 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:54:32,025 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-10-27 23:54:32,025 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:54:32,025 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:54:32,025 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:54:32,026 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:54:32,026 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:54:32,026 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:54:32,026 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:32,026 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:54:32,027 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:32,027 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:32,027 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:32,027 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:54:32,027 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:32,027 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:32,028 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:32,028 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:54:32,028 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:54:32,028 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:32,028 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:32,029 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:32,029 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:54:32,029 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:32,029 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:32,029 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:32,029 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:54:32,030 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:32,030 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:32,030 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,030 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,031 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:32,031 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,031 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,031 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:32,031 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:32,031 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:32,032 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:32,032 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,032 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN 2023-10-27 23:54:32,032 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:32,032 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:54:32,032 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,032 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,033 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,033 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:32,033 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:32,033 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 9 msec 2023-10-27 23:54:32,034 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:54:32,034 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:54:32,034 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450902034 2023-10-27 23:54:32,036 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,036 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:54:32,038 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:54:32,039 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=24b4632255f0f3b88a079696a3d55a55, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:54:32,039 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:54:32,039 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450932039 2023-10-27 23:54:32,039 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450992039 2023-10-27 23:54:32,039 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:54:32,042 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:54:32,044 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:54:32,044 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,044 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:54:32,044 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:54:32,289 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(454): Restart 6 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,290 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:54:32,290 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:54:32,290 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:54:32,290 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:54:32,290 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:54:32,290 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:54:32,290 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:54:32,290 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:54:32,291 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:54:32,292 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-10-27 23:54:32,292 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:54:32,292 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:54:32,292 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:54:32,292 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:54:32,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:54:32,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:54:32,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:32,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:54:32,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:32,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:32,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:32,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:54:32,293 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:32,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:32,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:32,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:54:32,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:54:32,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:32,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:32,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:32,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:54:32,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:32,294 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:32,295 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:32,295 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:54:32,295 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:32,295 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:32,295 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,295 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,295 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:32,296 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,296 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,296 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:32,296 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:32,296 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:32,296 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:32,297 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,297 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN 2023-10-27 23:54:32,297 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:32,297 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:54:32,297 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,297 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,297 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,297 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:32,298 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:32,298 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 6 msec 2023-10-27 23:54:32,298 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:54:32,299 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:54:32,299 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450902299 2023-10-27 23:54:32,301 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,301 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:54:32,303 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:54:32,303 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=24b4632255f0f3b88a079696a3d55a55, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:54:32,303 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:54:32,304 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450932304 2023-10-27 23:54:32,304 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450992304 2023-10-27 23:54:32,304 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:54:32,304 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:54:32,306 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450872306"},{"qualifier":"splitA","vlen":66,"tag":[],"timestamp":"1698450872306"},{"qualifier":"splitB","vlen":66,"tag":[],"timestamp":"1698450872306"}]},"ts":"1698450872306"} 2023-10-27 23:54:32,306 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1698450872306"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450872306"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450872306"}]},"ts":"1698450872306"} 2023-10-27 23:54:32,307 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1698450872306"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450872306"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450872306"}]},"ts":"1698450872306"} 2023-10-27 23:54:32,308 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:54:32,308 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,308 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:54:32,308 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:54:32,554 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(454): Restart 7 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,554 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:54:32,554 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:54:32,554 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:54:32,555 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:54:32,555 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:54:32,556 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:54:32,556 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:54:32,556 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:54:32,556 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:54:32,557 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10] 2023-10-27 23:54:32,557 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:54:32,557 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:54:32,558 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:54:32,558 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:54:32,558 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:54:32,558 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:54:32,558 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:32,558 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:54:32,558 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:32,558 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:32,559 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:32,559 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:54:32,559 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:32,559 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:32,559 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:32,560 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:54:32,560 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:54:32,560 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:32,560 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:32,560 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:32,560 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:54:32,560 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:32,560 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:32,560 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:32,561 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:54:32,561 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:32,561 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:32,561 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,561 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,561 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:32,562 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,562 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,562 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:32,562 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:32,562 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:32,562 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:32,562 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,562 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN 2023-10-27 23:54:32,562 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:32,563 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:54:32,563 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,563 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,563 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,563 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:32,563 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:32,563 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 6 msec 2023-10-27 23:54:32,564 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:54:32,564 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:54:32,565 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450902565 2023-10-27 23:54:32,566 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:54:32,566 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,568 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:54:32,568 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=24b4632255f0f3b88a079696a3d55a55, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:54:32,568 INFO [Listener at localhost/43373] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:32,569 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=6def62902b0889de037790b4aa137091, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-10-27 23:54:32,569 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=00f37dd06d8c5cff5d237ebb3d715040, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-10-27 23:54:32,569 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:54:32,569 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450932569 2023-10-27 23:54:32,569 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450992569 2023-10-27 23:54:32,569 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:54:32,569 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1698450872569"},{"qualifier":"splitA","vlen":66,"tag":[],"timestamp":"1698450872569"},{"qualifier":"splitB","vlen":66,"tag":[],"timestamp":"1698450872569"}]},"ts":"1698450872569"} 2023-10-27 23:54:32,569 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1698450872569"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450872569"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450872569"}]},"ts":"1698450872569"} 2023-10-27 23:54:32,569 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1698450872569"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450872569"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450872569"}]},"ts":"1698450872569"} 2023-10-27 23:54:32,571 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:54:32,573 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:54:32,573 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,573 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:54:32,573 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:54:32,819 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(454): Restart 8 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,819 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:54:32,819 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:54:32,820 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:54:32,820 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:54:32,820 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:54:32,820 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:54:32,820 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:54:32,820 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:54:32,821 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:54:32,822 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11] 2023-10-27 23:54:32,822 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:54:32,822 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:54:32,822 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:54:32,823 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:54:32,823 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:54:32,823 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:54:32,823 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:32,823 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:54:32,823 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:32,823 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:32,824 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:32,824 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:54:32,824 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:32,824 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:32,824 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:32,824 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:54:32,824 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:54:32,824 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:32,824 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:32,825 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:32,825 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:54:32,825 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:32,825 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:32,825 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:32,825 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:54:32,826 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:32,826 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:32,826 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,826 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,826 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:32,826 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,826 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:32,827 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:32,827 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:32,827 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:32,827 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:32,827 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,828 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN 2023-10-27 23:54:32,828 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:32,828 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:54:32,828 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,828 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,828 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,828 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:32,828 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:32,828 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 7 msec 2023-10-27 23:54:32,829 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:54:32,830 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:54:32,830 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450902830 2023-10-27 23:54:32,832 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:54:32,832 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,834 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:54:32,834 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=24b4632255f0f3b88a079696a3d55a55, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:54:32,835 INFO [Listener at localhost/43373] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:32,835 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=6def62902b0889de037790b4aa137091, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-10-27 23:54:32,835 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=00f37dd06d8c5cff5d237ebb3d715040, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-10-27 23:54:32,835 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:54:32,835 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450932835 2023-10-27 23:54:32,835 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450992835 2023-10-27 23:54:32,835 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 3 msec 2023-10-27 23:54:32,835 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:54:32,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=6def62902b0889de037790b4aa137091, ASSIGN}, {pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=00f37dd06d8c5cff5d237ebb3d715040, ASSIGN}] 2023-10-27 23:54:32,838 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:54:32,838 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:32,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:54:32,838 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:54:33,069 ERROR [BP-446492043-188.40.62.62-1698450761445 heartbeating to localhost/127.0.0.1:46525] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-446492043-188.40.62.62-1698450761445 (Datanode Uuid 41efbad2-7a69-4304-ba15-0342fa6d22c3) service to localhost/127.0.0.1:46525 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-10-27 23:54:33,085 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(454): Restart 9 exec state=pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,085 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:54:33,086 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:54:33,086 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:54:33,086 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:54:33,086 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:54:33,086 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:54:33,086 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:54:33,086 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:54:33,087 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:54:33,088 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11, 12] 2023-10-27 23:54:33,088 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:54:33,088 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:54:33,088 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:54:33,089 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:54:33,089 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:54:33,089 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:54:33,089 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:33,089 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:54:33,089 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:33,089 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:33,089 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:33,089 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:54:33,090 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:33,090 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:33,090 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:33,090 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:54:33,090 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:54:33,090 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:33,090 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:33,090 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:33,090 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:54:33,091 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:33,091 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:33,091 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:33,091 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:54:33,091 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:33,091 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:33,091 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:33,091 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:33,092 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:33,092 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:33,092 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:33,092 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:33,092 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:33,092 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:33,092 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:33,093 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,093 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN 2023-10-27 23:54:33,093 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:33,093 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:54:33,093 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,093 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,093 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,093 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:33,093 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:33,093 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 6 msec 2023-10-27 23:54:33,094 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:54:33,095 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:54:33,095 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450903095 2023-10-27 23:54:33,096 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,096 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:54:33,099 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:54:33,100 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=24b4632255f0f3b88a079696a3d55a55, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:54:33,100 INFO [Listener at localhost/43373] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:33,100 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=6def62902b0889de037790b4aa137091, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-10-27 23:54:33,100 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=00f37dd06d8c5cff5d237ebb3d715040, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-10-27 23:54:33,100 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:54:33,100 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450933100 2023-10-27 23:54:33,100 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450993100 2023-10-27 23:54:33,101 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 4 msec 2023-10-27 23:54:33,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=6def62902b0889de037790b4aa137091, ASSIGN}, {pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=00f37dd06d8c5cff5d237ebb3d715040, ASSIGN}] 2023-10-27 23:54:33,101 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:54:33,103 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=00f37dd06d8c5cff5d237ebb3d715040, ASSIGN 2023-10-27 23:54:33,104 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=00f37dd06d8c5cff5d237ebb3d715040, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,43641,1698450767195; forceNewPlan=false, retain=false 2023-10-27 23:54:33,105 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=6def62902b0889de037790b4aa137091, ASSIGN 2023-10-27 23:54:33,105 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=6def62902b0889de037790b4aa137091, ASSIGN; state=CLOSED, location=jenkins-hbase16.apache.org,43641,1698450767195; forceNewPlan=false, retain=false 2023-10-27 23:54:33,254 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-10-27 23:54:33,256 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=115 updating hbase:meta row=6def62902b0889de037790b4aa137091, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:33,256 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1698450873256"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450873256"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450873256"}]},"ts":"1698450873256"} 2023-10-27 23:54:33,257 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=117, ppid=115, state=RUNNABLE; OpenRegionProcedure 6def62902b0889de037790b4aa137091, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:54:33,259 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=116 updating hbase:meta row=00f37dd06d8c5cff5d237ebb3d715040, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:33,259 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1698450873259"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450873259"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450873259"}]},"ts":"1698450873259"} 2023-10-27 23:54:33,260 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=118, ppid=116, state=RUNNABLE; OpenRegionProcedure 00f37dd06d8c5cff5d237ebb3d715040, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:54:33,409 DEBUG [RSProcedureDispatcher-pool-3] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:33,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091. 2023-10-27 23:54:33,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6def62902b0889de037790b4aa137091, NAME => 'testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.', STARTKEY => '', ENDKEY => '41'} 2023-10-27 23:54:33,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:54:33,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution 6def62902b0889de037790b4aa137091 2023-10-27 23:54:33,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:33,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 6def62902b0889de037790b4aa137091 2023-10-27 23:54:33,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 6def62902b0889de037790b4aa137091 2023-10-27 23:54:33,413 INFO [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 6def62902b0889de037790b4aa137091 2023-10-27 23:54:33,414 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1 2023-10-27 23:54:33,414 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1 2023-10-27 23:54:33,414 INFO [StoreOpener-6def62902b0889de037790b4aa137091-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6def62902b0889de037790b4aa137091 columnFamilyName cf1 2023-10-27 23:54:33,428 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4-bottom 2023-10-27 23:54:33,432 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb-bottom 2023-10-27 23:54:33,436 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f-bottom 2023-10-27 23:54:33,439 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207-bottom 2023-10-27 23:54:33,443 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8-bottom 2023-10-27 23:54:33,448 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7-bottom 2023-10-27 23:54:33,453 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf-bottom 2023-10-27 23:54:33,453 INFO [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(310): Store=6def62902b0889de037790b4aa137091/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:33,453 INFO [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 6def62902b0889de037790b4aa137091 2023-10-27 23:54:33,454 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2 2023-10-27 23:54:33,454 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2 2023-10-27 23:54:33,454 INFO [StoreOpener-6def62902b0889de037790b4aa137091-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6def62902b0889de037790b4aa137091 columnFamilyName cf2 2023-10-27 23:54:33,468 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf-bottom 2023-10-27 23:54:33,479 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4-bottom 2023-10-27 23:54:33,483 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4-bottom 2023-10-27 23:54:33,488 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a-bottom 2023-10-27 23:54:33,492 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3-bottom 2023-10-27 23:54:33,496 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2-bottom 2023-10-27 23:54:33,501 DEBUG [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741-bottom 2023-10-27 23:54:33,501 INFO [StoreOpener-6def62902b0889de037790b4aa137091-1] regionserver.HStore(310): Store=6def62902b0889de037790b4aa137091/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:33,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091 2023-10-27 23:54:33,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091 2023-10-27 23:54:33,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:54:33,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 6def62902b0889de037790b4aa137091 2023-10-27 23:54:33,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 6def62902b0889de037790b4aa137091; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69894832, jitterRate=0.04151415824890137}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:54:33,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 6def62902b0889de037790b4aa137091: 2023-10-27 23:54:33,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091., pid=117, masterSystemTime=1698450873409 2023-10-27 23:54:33,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:54:33,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:54:33,507 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:54:33,507 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:54:33,507 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091. 2023-10-27 23:54:33,507 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091. 2023-10-27 23:54:33,507 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1912): 6def62902b0889de037790b4aa137091/cf1 is initiating minor compaction (all files) 2023-10-27 23:54:33,507 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1912): 6def62902b0889de037790b4aa137091/cf2 is initiating minor compaction (all files) 2023-10-27 23:54:33,507 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6def62902b0889de037790b4aa137091/cf1 in testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091. 2023-10-27 23:54:33,507 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6def62902b0889de037790b4aa137091/cf2 in testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091. 2023-10-27 23:54:33,508 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2-bottom] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/.tmp, totalSize=34.9 K 2023-10-27 23:54:33,508 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf-bottom] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/.tmp, totalSize=34.9 K 2023-10-27 23:54:33,508 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450868542 2023-10-27 23:54:33,508 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450868542 2023-10-27 23:54:33,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091. 2023-10-27 23:54:33,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091. 2023-10-27 23:54:33,509 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450868799 2023-10-27 23:54:33,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040. 2023-10-27 23:54:33,509 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=115 updating hbase:meta row=6def62902b0889de037790b4aa137091, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:33,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 00f37dd06d8c5cff5d237ebb3d715040, NAME => 'testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.', STARTKEY => '41', ENDKEY => ''} 2023-10-27 23:54:33,509 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450868799 2023-10-27 23:54:33,509 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1698450873509"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450873509"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450873509"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450873509"}]},"ts":"1698450873509"} 2023-10-27 23:54:33,509 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450869010 2023-10-27 23:54:33,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:54:33,509 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450869010 2023-10-27 23:54:33,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution 00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,509 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450869252 2023-10-27 23:54:33,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:33,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,510 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450869469 2023-10-27 23:54:33,510 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450869252 2023-10-27 23:54:33,510 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450869694 2023-10-27 23:54:33,510 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450869469 2023-10-27 23:54:33,511 INFO [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,511 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1698450869941 2023-10-27 23:54:33,511 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450869694 2023-10-27 23:54:33,511 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1698450869941 2023-10-27 23:54:33,512 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1 2023-10-27 23:54:33,512 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1 2023-10-27 23:54:33,512 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=117, resume processing ppid=115 2023-10-27 23:54:33,512 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=117, ppid=115, state=SUCCESS; OpenRegionProcedure 6def62902b0889de037790b4aa137091, server=jenkins-hbase16.apache.org,43641,1698450767195 in 253 msec 2023-10-27 23:54:33,512 INFO [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 00f37dd06d8c5cff5d237ebb3d715040 columnFamilyName cf1 2023-10-27 23:54:33,513 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=6def62902b0889de037790b4aa137091, ASSIGN in 411 msec 2023-10-27 23:54:33,528 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] throttle.PressureAwareThroughputController(145): 6def62902b0889de037790b4aa137091#cf2#compaction#109 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:54:33,528 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6def62902b0889de037790b4aa137091#cf1#compaction#108 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:54:33,532 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4-top 2023-10-27 23:54:33,537 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb-top 2023-10-27 23:54:33,540 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f-top 2023-10-27 23:54:33,541 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/.tmp/cf2/ce1d4891670d4fedb40878d15cfba8ae as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/ce1d4891670d4fedb40878d15cfba8ae 2023-10-27 23:54:33,542 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/.tmp/cf1/4cd0dbe4151646598e075aace160c5e9 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/4cd0dbe4151646598e075aace160c5e9 2023-10-27 23:54:33,546 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 6def62902b0889de037790b4aa137091/cf2 of 6def62902b0889de037790b4aa137091 into ce1d4891670d4fedb40878d15cfba8ae(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:54:33,546 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 6def62902b0889de037790b4aa137091/cf1 of 6def62902b0889de037790b4aa137091 into 4cd0dbe4151646598e075aace160c5e9(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:54:33,546 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6def62902b0889de037790b4aa137091: 2023-10-27 23:54:33,546 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6def62902b0889de037790b4aa137091: 2023-10-27 23:54:33,546 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091., storeName=6def62902b0889de037790b4aa137091/cf2, priority=9, startTime=1698450873507; duration=0sec 2023-10-27 23:54:33,546 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091., storeName=6def62902b0889de037790b4aa137091/cf1, priority=9, startTime=1698450873506; duration=0sec 2023-10-27 23:54:33,546 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:33,546 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:33,547 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207-top 2023-10-27 23:54:33,550 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8-top 2023-10-27 23:54:33,553 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7-top 2023-10-27 23:54:33,556 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf-top 2023-10-27 23:54:33,557 INFO [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(310): Store=00f37dd06d8c5cff5d237ebb3d715040/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:33,557 INFO [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,557 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2 2023-10-27 23:54:33,558 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2 2023-10-27 23:54:33,558 INFO [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 00f37dd06d8c5cff5d237ebb3d715040 columnFamilyName cf2 2023-10-27 23:54:33,568 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf-top 2023-10-27 23:54:33,571 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4-top 2023-10-27 23:54:33,575 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4-top 2023-10-27 23:54:33,577 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a-top 2023-10-27 23:54:33,581 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3-top 2023-10-27 23:54:33,584 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2-top 2023-10-27 23:54:33,587 DEBUG [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741-top 2023-10-27 23:54:33,587 INFO [StoreOpener-00f37dd06d8c5cff5d237ebb3d715040-1] regionserver.HStore(310): Store=00f37dd06d8c5cff5d237ebb3d715040/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:33,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:54:33,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,592 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 00f37dd06d8c5cff5d237ebb3d715040; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75453462, jitterRate=0.12434419989585876}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:54:33,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 00f37dd06d8c5cff5d237ebb3d715040: 2023-10-27 23:54:33,592 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040., pid=118, masterSystemTime=1698450873409 2023-10-27 23:54:33,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:33,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:54:33,593 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:54:33,593 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:54:33,593 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040. 2023-10-27 23:54:33,594 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1912): 00f37dd06d8c5cff5d237ebb3d715040/cf1 is initiating minor compaction (all files) 2023-10-27 23:54:33,594 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040. 2023-10-27 23:54:33,594 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1912): 00f37dd06d8c5cff5d237ebb3d715040/cf2 is initiating minor compaction (all files) 2023-10-27 23:54:33,594 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00f37dd06d8c5cff5d237ebb3d715040/cf1 in testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040. 2023-10-27 23:54:33,594 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00f37dd06d8c5cff5d237ebb3d715040/cf2 in testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040. 2023-10-27 23:54:33,594 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2-top] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/.tmp, totalSize=34.9 K 2023-10-27 23:54:33,594 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf-top] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/.tmp, totalSize=34.9 K 2023-10-27 23:54:33,595 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1698450868542 2023-10-27 23:54:33,595 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1698450868542 2023-10-27 23:54:33,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040. 2023-10-27 23:54:33,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040. 2023-10-27 23:54:33,595 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1698450868799 2023-10-27 23:54:33,595 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1698450868799 2023-10-27 23:54:33,595 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1698450869010 2023-10-27 23:54:33,595 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=116 updating hbase:meta row=00f37dd06d8c5cff5d237ebb3d715040, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:33,595 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1698450873595"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450873595"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450873595"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450873595"}]},"ts":"1698450873595"} 2023-10-27 23:54:33,595 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1698450869010 2023-10-27 23:54:33,596 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1698450869252 2023-10-27 23:54:33,596 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1698450869469 2023-10-27 23:54:33,596 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1698450869252 2023-10-27 23:54:33,596 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1698450869694 2023-10-27 23:54:33,596 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1698450869469 2023-10-27 23:54:33,597 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1698450869941 2023-10-27 23:54:33,597 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1698450869694 2023-10-27 23:54:33,597 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1698450869941 2023-10-27 23:54:33,598 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=118, resume processing ppid=116 2023-10-27 23:54:33,598 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=118, ppid=116, state=SUCCESS; OpenRegionProcedure 00f37dd06d8c5cff5d237ebb3d715040, server=jenkins-hbase16.apache.org,43641,1698450767195 in 337 msec 2023-10-27 23:54:33,600 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=116, resume processing ppid=112 2023-10-27 23:54:33,600 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=00f37dd06d8c5cff5d237ebb3d715040, ASSIGN in 497 msec 2023-10-27 23:54:33,600 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-10-27 23:54:33,600 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1758): TESTING: Kill BEFORE store update: pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,600 INFO [PEWorker-1] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:54:33,600 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1979): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1702) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1391) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1963) 2023-10-27 23:54:33,601 INFO [Listener at localhost/43373] procedure.MasterProcedureTestingUtility(454): Restart 10 exec state=pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,601 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-10-27 23:54:33,601 INFO [Listener at localhost/43373] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:54:33,601 INFO [Listener at localhost/43373] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:54:33,602 ERROR [Listener at localhost/43373] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-10-27 23:54:33,602 INFO [Listener at localhost/43373] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-10-27 23:54:33,602 INFO [Listener at localhost/43373] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-10-27 23:54:33,602 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(561): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-10-27 23:54:33,602 INFO [Listener at localhost/43373] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-10-27 23:54:33,603 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(581): Recovered RegionProcedureStore lease in 0 msec 2023-10-27 23:54:33,604 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11, 12, 13] 2023-10-27 23:54:33,604 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-10-27 23:54:33,604 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-10-27 23:54:33,604 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=115, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[15, 16, 21] 2023-10-27 23:54:33,605 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=117, ppid=115, class=org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure) stack ids=[17, 20] 2023-10-27 23:54:33,605 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=116, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[14, 18, 23] 2023-10-27 23:54:33,605 DEBUG [Listener at localhost/43373] store.ProcedureTree(114): Procedure Procedure(pid=118, ppid=116, class=org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure) stack ids=[19, 22] 2023-10-27 23:54:33,605 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-10-27 23:54:33,605 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-10-27 23:54:33,605 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-10-27 23:54:33,606 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-10-27 23:54:33,606 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:33,606 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] throttle.PressureAwareThroughputController(145): 00f37dd06d8c5cff5d237ebb3d715040#cf2#compaction#110 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:54:33,606 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1a178e5e9b9e5a90d2396ab013c5750a, daughterA=8285b45cae27573bc22fb00c23a6d6fe, daughterB=9aca977d0db042f340c741ee9446ec83 2023-10-27 23:54:33,606 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:33,606 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00f37dd06d8c5cff5d237ebb3d715040#cf1#compaction#111 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:54:33,607 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-10-27 23:54:33,607 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:33,607 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=a6d7da66a8a7b852f46fc7477e45d6ab, daughterA=409ad364226a99ff4e85bbe38f51653a, daughterB=52ad738b07f186a78f22431d82fc4c1f 2023-10-27 23:54:33,607 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:33,607 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-10-27 23:54:33,608 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:33,608 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=38d29e3b4a5d0b766af1d906cf63bcf5, daughterA=a03293152823ec15f91cc2b99007ad83, daughterB=93f423a1d296125aea0b5218045bfe46 2023-10-27 23:54:33,608 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=38d29e3b4a5d0b766af1d906cf63bcf5, ASSIGN 2023-10-27 23:54:33,608 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:33,608 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-10-27 23:54:33,609 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:33,609 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=aa542793a0b70c04a27ad5d5b58f9b15, daughterA=c6d17e2577f36e5cbf594fca675240be, daughterB=433bacad157e4d0e8f716e1942033cb2 2023-10-27 23:54:33,609 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:33,609 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-10-27 23:54:33,609 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:33,609 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=3087a119cbcd659122571a2b31e50ec2, daughterA=f0da177e9a77436613e559466a9befb3, daughterB=f3419eb3ecb19fbd75b9ef498fab9b83 2023-10-27 23:54:33,609 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:33,610 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-10-27 23:54:33,610 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:33,610 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:33,610 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRollbackForSplitTableRegionWithReplica, region=420eeb15f6ee5bc27b6201436916974b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=b2d34fd5cbfd39a1be2f8cf0f5dbe83a, daughterA=e3811836f6abe95770b07edc19532636, daughterB=7d166278a57d7fa9da33b78d228288d6 2023-10-27 23:54:33,610 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:33,611 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-10-27 23:54:33,611 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:33,611 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:33,611 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-10-27 23:54:33,611 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(410): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:33,613 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,613 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN 2023-10-27 23:54:33,613 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:33,613 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=6def62902b0889de037790b4aa137091, ASSIGN 2023-10-27 23:54:33,613 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=00f37dd06d8c5cff5d237ebb3d715040, ASSIGN 2023-10-27 23:54:33,613 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=117, ppid=115, state=SUCCESS; OpenRegionProcedure 6def62902b0889de037790b4aa137091, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:33,613 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(467): Loading pid=118, ppid=116, state=SUCCESS; OpenRegionProcedure 00f37dd06d8c5cff5d237ebb3d715040, server=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:33,613 DEBUG [Listener at localhost/43373] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 held the lock before restarting, call acquireLock to restore it. 2023-10-27 23:54:33,614 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,614 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,614 INFO [Listener at localhost/43373] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,614 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:33,614 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure 24b4632255f0f3b88a079696a3d55a55, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:33,614 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=6def62902b0889de037790b4aa137091, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:33,614 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=00f37dd06d8c5cff5d237ebb3d715040, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:33,614 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=117, ppid=115, state=SUCCESS; OpenRegionProcedure 6def62902b0889de037790b4aa137091, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:33,614 DEBUG [Listener at localhost/43373] procedure2.Procedure(946): pid=118, ppid=116, state=SUCCESS; OpenRegionProcedure 00f37dd06d8c5cff5d237ebb3d715040, server=jenkins-hbase16.apache.org,43641,1698450767195 didn't hold the lock before restarting, skip acquiring lock. 2023-10-27 23:54:33,614 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(595): Loaded RegionProcedureStore in 11 msec 2023-10-27 23:54:33,616 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-10-27 23:54:33,616 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase16.apache.org,36599,1698450767241, table=hbase:meta, region=1588230740 2023-10-27 23:54:33,633 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1399): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:33,633 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1698450903633 2023-10-27 23:54:33,633 DEBUG [Listener at localhost/43373] assignment.AssignmentManager(1527): Joining cluster... 2023-10-27 23:54:33,637 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=5005e44f9940b2c7d862a285ee6d8687, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,39059,1698450767147, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147, openSeqNum=2 2023-10-27 23:54:33,638 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=24b4632255f0f3b88a079696a3d55a55, regionState=CLOSED, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=2 2023-10-27 23:54:33,638 INFO [Listener at localhost/43373] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase16.apache.org,43641,1698450767195, table=testRecoveryAndDoubleExecution, region=24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:33,638 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=6def62902b0889de037790b4aa137091, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=86 2023-10-27 23:54:33,638 DEBUG [Listener at localhost/43373] assignment.RegionStateStore(147): Load hbase:meta entry region=00f37dd06d8c5cff5d237ebb3d715040, regionState=OPEN, lastHost=jenkins-hbase16.apache.org,43641,1698450767195, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195, openSeqNum=86 2023-10-27 23:54:33,638 INFO [Listener at localhost/43373] assignment.AssignmentManager(1539): Number of RegionServers=3 2023-10-27 23:54:33,638 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1698450933638 2023-10-27 23:54:33,638 INFO [Listener at localhost/43373] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1698450993638 2023-10-27 23:54:33,639 INFO [Listener at localhost/43373] assignment.AssignmentManager(1546): Joined the cluster in 5 msec 2023-10-27 23:54:33,639 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-10-27 23:54:33,641 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/.tmp/cf1/fd84c1eb44ca454a8be1bb6deb75e88c as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/fd84c1eb44ca454a8be1bb6deb75e88c 2023-10-27 23:54:33,642 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=112, state=SUCCESS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=24b4632255f0f3b88a079696a3d55a55, daughterA=6def62902b0889de037790b4aa137091, daughterB=00f37dd06d8c5cff5d237ebb3d715040 in 3.4400 sec 2023-10-27 23:54:33,645 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/.tmp/cf2/a32d6d3046c74ae99902083c01bee84c as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/a32d6d3046c74ae99902083c01bee84c 2023-10-27 23:54:33,646 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 00f37dd06d8c5cff5d237ebb3d715040/cf1 of 00f37dd06d8c5cff5d237ebb3d715040 into fd84c1eb44ca454a8be1bb6deb75e88c(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:54:33,646 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00f37dd06d8c5cff5d237ebb3d715040: 2023-10-27 23:54:33,646 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040., storeName=00f37dd06d8c5cff5d237ebb3d715040/cf1, priority=9, startTime=1698450873592; duration=0sec 2023-10-27 23:54:33,646 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:33,649 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 00f37dd06d8c5cff5d237ebb3d715040/cf2 of 00f37dd06d8c5cff5d237ebb3d715040 into a32d6d3046c74ae99902083c01bee84c(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:54:33,649 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00f37dd06d8c5cff5d237ebb3d715040: 2023-10-27 23:54:33,649 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040., storeName=00f37dd06d8c5cff5d237ebb3d715040/cf2, priority=9, startTime=1698450873593; duration=0sec 2023-10-27 23:54:33,649 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:33,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43641] ipc.CallRunner(144): callId: 866 service: ClientService methodName: Scan size: 154 connection: 188.40.62.62:48696 deadline: 1698450933890, exception=org.apache.hadoop.hbase.NotServingRegionException: testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55. is not online on jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:34,003 DEBUG [Listener at localhost/43373] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-10-27 23:54:34,009 INFO [Listener at localhost/43373] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-10-27 23:54:34,011 DEBUG [Listener at localhost/43373] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-10-27 23:54:34,025 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:54:34,026 INFO [Listener at localhost/43373] client.HBaseAdmin$15(890): Started disable of testRecoveryAndDoubleExecution 2023-10-27 23:54:34,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable testRecoveryAndDoubleExecution 2023-10-27 23:54:34,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=119, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:34,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=119 2023-10-27 23:54:34,030 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450874030"}]},"ts":"1698450874030"} 2023-10-27 23:54:34,030 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=DISABLING in hbase:meta 2023-10-27 23:54:34,071 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRecoveryAndDoubleExecution to state=DISABLING 2023-10-27 23:54:34,072 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=120, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=6def62902b0889de037790b4aa137091, UNASSIGN}, {pid=121, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=00f37dd06d8c5cff5d237ebb3d715040, UNASSIGN}] 2023-10-27 23:54:34,073 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=00f37dd06d8c5cff5d237ebb3d715040, UNASSIGN 2023-10-27 23:54:34,073 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=00f37dd06d8c5cff5d237ebb3d715040, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:34,073 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1698450874073"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450874073"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450874073"}]},"ts":"1698450874073"} 2023-10-27 23:54:34,075 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=122, ppid=121, state=RUNNABLE; CloseRegionProcedure 00f37dd06d8c5cff5d237ebb3d715040, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:54:34,077 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=120, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=6def62902b0889de037790b4aa137091, UNASSIGN 2023-10-27 23:54:34,077 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=120 updating hbase:meta row=6def62902b0889de037790b4aa137091, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:34,077 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1698450874077"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450874077"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450874077"}]},"ts":"1698450874077"} 2023-10-27 23:54:34,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=123, ppid=120, state=RUNNABLE; CloseRegionProcedure 6def62902b0889de037790b4aa137091, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:54:34,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=119 2023-10-27 23:54:34,226 DEBUG [RSProcedureDispatcher-pool-4] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:34,227 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:34,227 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 00f37dd06d8c5cff5d237ebb3d715040, disabling compactions & flushes 2023-10-27 23:54:34,227 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040. 2023-10-27 23:54:34,227 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040. 2023-10-27 23:54:34,227 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040. after waiting 0 ms 2023-10-27 23:54:34,227 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040. 2023-10-27 23:54:34,228 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf-top] to archive 2023-10-27 23:54:34,228 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:54:34,230 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,231 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,232 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,234 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,235 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,236 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,237 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,238 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2-top] to archive 2023-10-27 23:54:34,238 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:54:34,239 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,240 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,241 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,242 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,243 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,244 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,245 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-10-27 23:54:34,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:34,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040. 2023-10-27 23:54:34,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 00f37dd06d8c5cff5d237ebb3d715040: 2023-10-27 23:54:34,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:34,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 6def62902b0889de037790b4aa137091 2023-10-27 23:54:34,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 6def62902b0889de037790b4aa137091, disabling compactions & flushes 2023-10-27 23:54:34,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091. 2023-10-27 23:54:34,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091. 2023-10-27 23:54:34,250 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=00f37dd06d8c5cff5d237ebb3d715040, regionState=CLOSED 2023-10-27 23:54:34,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091. after waiting 0 ms 2023-10-27 23:54:34,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091. 2023-10-27 23:54:34,251 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1698450874250"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450874250"}]},"ts":"1698450874250"} 2023-10-27 23:54:34,252 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf-bottom] to archive 2023-10-27 23:54:34,252 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:54:34,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=122, resume processing ppid=121 2023-10-27 23:54:34,254 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=122, ppid=121, state=SUCCESS; CloseRegionProcedure 00f37dd06d8c5cff5d237ebb3d715040, server=jenkins-hbase16.apache.org,43641,1698450767195 in 177 msec 2023-10-27 23:54:34,254 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,255 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=121, ppid=119, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=00f37dd06d8c5cff5d237ebb3d715040, UNASSIGN in 182 msec 2023-10-27 23:54:34,255 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,256 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,257 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,257 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,258 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,259 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,261 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2-bottom] to archive 2023-10-27 23:54:34,261 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:54:34,262 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,263 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,264 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,265 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,266 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,266 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,267 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-10-27 23:54:34,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:34,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091. 2023-10-27 23:54:34,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 6def62902b0889de037790b4aa137091: 2023-10-27 23:54:34,273 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 6def62902b0889de037790b4aa137091 2023-10-27 23:54:34,273 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=120 updating hbase:meta row=6def62902b0889de037790b4aa137091, regionState=CLOSED 2023-10-27 23:54:34,273 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1698450874273"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450874273"}]},"ts":"1698450874273"} 2023-10-27 23:54:34,275 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=123, resume processing ppid=120 2023-10-27 23:54:34,275 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=123, ppid=120, state=SUCCESS; CloseRegionProcedure 6def62902b0889de037790b4aa137091, server=jenkins-hbase16.apache.org,43641,1698450767195 in 196 msec 2023-10-27 23:54:34,276 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=120, resume processing ppid=119 2023-10-27 23:54:34,276 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=120, ppid=119, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=6def62902b0889de037790b4aa137091, UNASSIGN in 203 msec 2023-10-27 23:54:34,277 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450874277"}]},"ts":"1698450874277"} 2023-10-27 23:54:34,278 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=DISABLED in hbase:meta 2023-10-27 23:54:34,288 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRecoveryAndDoubleExecution to state=DISABLED 2023-10-27 23:54:34,289 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=119, state=SUCCESS; DisableTableProcedure table=testRecoveryAndDoubleExecution in 261 msec 2023-10-27 23:54:34,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=119 2023-10-27 23:54:34,332 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRecoveryAndDoubleExecution, procId: 119 completed 2023-10-27 23:54:34,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete testRecoveryAndDoubleExecution 2023-10-27 23:54:34,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=124, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:34,335 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=124, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:34,336 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=124, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:34,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=124 2023-10-27 23:54:34,339 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,339 DEBUG [HFileArchiver-15] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:34,339 DEBUG [HFileArchiver-14] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091 2023-10-27 23:54:34,342 DEBUG [HFileArchiver-15] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/recovered.edits] 2023-10-27 23:54:34,342 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/recovered.edits] 2023-10-27 23:54:34,342 DEBUG [HFileArchiver-14] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/recovered.edits] 2023-10-27 23:54:34,346 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/.regioninfo to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/.regioninfo 2023-10-27 23:54:34,347 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/.regioninfo to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/.regioninfo 2023-10-27 23:54:34,347 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/1b440e72bfc24900bf116a81845193a4 2023-10-27 23:54:34,348 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/46a9bf3709a942f5867e091b8f085cbb 2023-10-27 23:54:34,349 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,349 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/51f737396e99468cbd1e87a1c8088e6f 2023-10-27 23:54:34,350 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/1b440e72bfc24900bf116a81845193a4.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,350 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,492 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/5da2fdb2de0741f5a19326bd3a22c207 2023-10-27 23:54:34,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=124 2023-10-27 23:54:34,493 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/46a9bf3709a942f5867e091b8f085cbb.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,494 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/a0f5d933fe754195aed6fca431e18ce8 2023-10-27 23:54:34,494 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,495 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/51f737396e99468cbd1e87a1c8088e6f.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,496 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/ab6cf8f8528e4010ac620294cd4a93a7 2023-10-27 23:54:34,496 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,496 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/5da2fdb2de0741f5a19326bd3a22c207.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,497 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,497 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf1/f62fe57988ed466b844ebb088bdd1fdf 2023-10-27 23:54:34,497 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/a0f5d933fe754195aed6fca431e18ce8.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,498 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,498 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/ab6cf8f8528e4010ac620294cd4a93a7.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,499 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,499 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf1/f62fe57988ed466b844ebb088bdd1fdf.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,500 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/19cd9f6c45444674ad42782a1dcfbeaf 2023-10-27 23:54:34,501 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/94921a45fbff4ff28109056240e59ec4 2023-10-27 23:54:34,501 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,502 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/19cd9f6c45444674ad42782a1dcfbeaf.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,502 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/a78b6261d9f0484790e834172b9147a4 2023-10-27 23:54:34,503 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,503 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/94921a45fbff4ff28109056240e59ec4.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,503 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/df0c477557384f79a606523bb4f58f8a 2023-10-27 23:54:34,504 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,504 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/e2137e2a2fbe4f239761d6c3b26491c3 2023-10-27 23:54:34,505 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/a78b6261d9f0484790e834172b9147a4.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,505 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,505 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/eca3177bff2f41caa13763f4b034c8e2 2023-10-27 23:54:34,506 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/df0c477557384f79a606523bb4f58f8a.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,506 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,506 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/cf2/fb908c093a2e4a2495e37ed55e98e741 2023-10-27 23:54:34,507 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/e2137e2a2fbe4f239761d6c3b26491c3.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,507 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,507 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/eca3177bff2f41caa13763f4b034c8e2.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,508 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/6def62902b0889de037790b4aa137091/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,508 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/00f37dd06d8c5cff5d237ebb3d715040/cf2/fb908c093a2e4a2495e37ed55e98e741.24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,508 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/recovered.edits/85.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55/recovered.edits/85.seqid 2023-10-27 23:54:34,509 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/24b4632255f0f3b88a079696a3d55a55 2023-10-27 23:54:34,510 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/4cd0dbe4151646598e075aace160c5e9 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf1/4cd0dbe4151646598e075aace160c5e9 2023-10-27 23:54:34,510 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/fd84c1eb44ca454a8be1bb6deb75e88c to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf1/fd84c1eb44ca454a8be1bb6deb75e88c 2023-10-27 23:54:34,511 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/ce1d4891670d4fedb40878d15cfba8ae to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/cf2/ce1d4891670d4fedb40878d15cfba8ae 2023-10-27 23:54:34,512 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/a32d6d3046c74ae99902083c01bee84c to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/cf2/a32d6d3046c74ae99902083c01bee84c 2023-10-27 23:54:34,513 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/recovered.edits/90.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091/recovered.edits/90.seqid 2023-10-27 23:54:34,513 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/recovered.edits/90.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040/recovered.edits/90.seqid 2023-10-27 23:54:34,514 DEBUG [HFileArchiver-14] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/6def62902b0889de037790b4aa137091 2023-10-27 23:54:34,514 DEBUG [HFileArchiver-15] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testRecoveryAndDoubleExecution/00f37dd06d8c5cff5d237ebb3d715040 2023-10-27 23:54:34,514 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRecoveryAndDoubleExecution regions 2023-10-27 23:54:34,516 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=124, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:34,517 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testRecoveryAndDoubleExecution from hbase:meta 2023-10-27 23:54:34,520 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRecoveryAndDoubleExecution' descriptor. 2023-10-27 23:54:34,521 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=124, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:34,521 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRecoveryAndDoubleExecution' from region states. 2023-10-27 23:54:34,521 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450874521"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:34,521 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450874521"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:34,522 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450874521"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:34,523 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-10-27 23:54:34,523 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 24b4632255f0f3b88a079696a3d55a55, NAME => 'testRecoveryAndDoubleExecution,,1698450868039.24b4632255f0f3b88a079696a3d55a55.', STARTKEY => '', ENDKEY => '', OFFLINE => true, SPLIT => true}, {ENCODED => 6def62902b0889de037790b4aa137091, NAME => 'testRecoveryAndDoubleExecution,,1698450870199.6def62902b0889de037790b4aa137091.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => 00f37dd06d8c5cff5d237ebb3d715040, NAME => 'testRecoveryAndDoubleExecution,41,1698450870199.00f37dd06d8c5cff5d237ebb3d715040.', STARTKEY => '41', ENDKEY => ''}] 2023-10-27 23:54:34,523 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRecoveryAndDoubleExecution' as deleted. 2023-10-27 23:54:34,523 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1698450874523"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:34,525 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRecoveryAndDoubleExecution state from META 2023-10-27 23:54:34,533 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=124, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-10-27 23:54:34,534 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=124, state=SUCCESS; DeleteTableProcedure table=testRecoveryAndDoubleExecution in 200 msec 2023-10-27 23:54:34,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=124 2023-10-27 23:54:34,695 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRecoveryAndDoubleExecution, procId: 124 completed 2023-10-27 23:54:34,707 INFO [Listener at localhost/43373] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRecoveryAndDoubleExecution Thread=379 (was 370) - Thread LEAK? -, OpenFileDescriptor=841 (was 742) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=224 (was 225), ProcessCount=170 (was 171), AvailableMemoryMB=5402 (was 5466) 2023-10-27 23:54:34,718 INFO [Listener at localhost/43373] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionEmptyDaughter Thread=379, OpenFileDescriptor=841, MaxFileDescriptor=60000, SystemLoadAverage=224, ProcessCount=170, AvailableMemoryMB=5402 2023-10-27 23:54:34,718 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:54:34,729 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-10-27 23:54:34,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(492): Client=jenkins//188.40.62.62 set balanceSwitch=false 2023-10-27 23:54:34,731 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=125, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionEmptyDaughter 2023-10-27 23:54:34,733 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-10-27 23:54:34,733 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-10-27 23:54:34,735 DEBUG [HFileArchiver-16] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:34,735 DEBUG [HFileArchiver-16] backup.HFileArchiver(153): Directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99 empty. 2023-10-27 23:54:34,736 DEBUG [HFileArchiver-16] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:34,736 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionEmptyDaughter regions 2023-10-27 23:54:34,745 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/.tabledesc/.tableinfo.0000000001 2023-10-27 23:54:34,746 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(7675): creating {ENCODED => dfe214ec55570794a9b9dc2a65a69d99, NAME => 'testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionEmptyDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp 2023-10-27 23:54:34,752 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:34,752 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1604): Closing dfe214ec55570794a9b9dc2a65a69d99, disabling compactions & flushes 2023-10-27 23:54:34,752 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:34,752 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:34,752 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. after waiting 0 ms 2023-10-27 23:54:34,752 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:34,752 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:34,752 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1558): Region close journal for dfe214ec55570794a9b9dc2a65a69d99: 2023-10-27 23:54:34,754 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-10-27 23:54:34,755 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450874755"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450874755"}]},"ts":"1698450874755"} 2023-10-27 23:54:34,756 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-10-27 23:54:34,756 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-10-27 23:54:34,756 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450874756"}]},"ts":"1698450874756"} 2023-10-27 23:54:34,757 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=ENABLING in hbase:meta 2023-10-27 23:54:34,774 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-10-27 23:54:34,775 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-27 23:54:34,775 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-10-27 23:54:34,775 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-10-27 23:54:34,775 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-10-27 23:54:34,775 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=dfe214ec55570794a9b9dc2a65a69d99, ASSIGN}] 2023-10-27 23:54:34,777 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=dfe214ec55570794a9b9dc2a65a69d99, ASSIGN 2023-10-27 23:54:34,778 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=dfe214ec55570794a9b9dc2a65a69d99, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,39059,1698450767147; forceNewPlan=false, retain=false 2023-10-27 23:54:34,928 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-10-27 23:54:34,929 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=dfe214ec55570794a9b9dc2a65a69d99, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:34,929 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450874929"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450874929"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450874929"}]},"ts":"1698450874929"} 2023-10-27 23:54:34,931 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=127, ppid=126, state=RUNNABLE; OpenRegionProcedure dfe214ec55570794a9b9dc2a65a69d99, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:54:35,082 DEBUG [RSProcedureDispatcher-pool-5] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,085 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:35,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => dfe214ec55570794a9b9dc2a65a69d99, NAME => 'testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:54:35,086 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:54:35,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:35,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:35,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:35,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:35,087 INFO [StoreOpener-dfe214ec55570794a9b9dc2a65a69d99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:35,088 DEBUG [StoreOpener-dfe214ec55570794a9b9dc2a65a69d99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1 2023-10-27 23:54:35,088 DEBUG [StoreOpener-dfe214ec55570794a9b9dc2a65a69d99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1 2023-10-27 23:54:35,089 INFO [StoreOpener-dfe214ec55570794a9b9dc2a65a69d99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dfe214ec55570794a9b9dc2a65a69d99 columnFamilyName cf1 2023-10-27 23:54:35,089 INFO [StoreOpener-dfe214ec55570794a9b9dc2a65a69d99-1] regionserver.HStore(310): Store=dfe214ec55570794a9b9dc2a65a69d99/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:35,089 INFO [StoreOpener-dfe214ec55570794a9b9dc2a65a69d99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:35,090 DEBUG [StoreOpener-dfe214ec55570794a9b9dc2a65a69d99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2 2023-10-27 23:54:35,090 DEBUG [StoreOpener-dfe214ec55570794a9b9dc2a65a69d99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2 2023-10-27 23:54:35,091 INFO [StoreOpener-dfe214ec55570794a9b9dc2a65a69d99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dfe214ec55570794a9b9dc2a65a69d99 columnFamilyName cf2 2023-10-27 23:54:35,091 INFO [StoreOpener-dfe214ec55570794a9b9dc2a65a69d99-1] regionserver.HStore(310): Store=dfe214ec55570794a9b9dc2a65a69d99/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:35,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:35,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:35,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:54:35,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:35,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-10-27 23:54:35,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened dfe214ec55570794a9b9dc2a65a69d99; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65946085, jitterRate=-0.017326757311820984}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:54:35,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for dfe214ec55570794a9b9dc2a65a69d99: 2023-10-27 23:54:35,097 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99., pid=127, masterSystemTime=1698450875082 2023-10-27 23:54:35,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:35,099 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:35,099 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=dfe214ec55570794a9b9dc2a65a69d99, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,099 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450875099"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450875099"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450875099"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450875099"}]},"ts":"1698450875099"} 2023-10-27 23:54:35,102 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=127, resume processing ppid=126 2023-10-27 23:54:35,102 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=127, ppid=126, state=SUCCESS; OpenRegionProcedure dfe214ec55570794a9b9dc2a65a69d99, server=jenkins-hbase16.apache.org,39059,1698450767147 in 170 msec 2023-10-27 23:54:35,103 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=126, resume processing ppid=125 2023-10-27 23:54:35,104 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=126, ppid=125, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=dfe214ec55570794a9b9dc2a65a69d99, ASSIGN in 327 msec 2023-10-27 23:54:35,104 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-10-27 23:54:35,104 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450875104"}]},"ts":"1698450875104"} 2023-10-27 23:54:35,105 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=ENABLED in hbase:meta 2023-10-27 23:54:35,116 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-10-27 23:54:35,118 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=125, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionEmptyDaughter in 387 msec 2023-10-27 23:54:35,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:35,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,240 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-10-27 23:54:35,240 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,240 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-10-27 23:54:35,240 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-10-27 23:54:35,241 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,241 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,249 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,249 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,249 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,249 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,250 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,250 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,251 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:35,251 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:35,251 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,251 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,251 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:35,251 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:35,251 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:35,251 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:35,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,252 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:35,252 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:35,252 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:35,252 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:35,252 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. started... 2023-10-27 23:54:35,252 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,252 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:35,253 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:35,253 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dfe214ec55570794a9b9dc2a65a69d99 2/2 column families, dataSize=120 B heapSize=928 B 2023-10-27 23:54:35,253 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:35,257 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,264 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/1e18611e034a4a77873e2c82048d05f7 2023-10-27 23:54:35,266 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,266 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,266 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:35,266 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,266 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:35,275 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/8a0a5cb16d414089b9f11d5750c6652d 2023-10-27 23:54:35,278 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/1e18611e034a4a77873e2c82048d05f7 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/1e18611e034a4a77873e2c82048d05f7 2023-10-27 23:54:35,282 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/1e18611e034a4a77873e2c82048d05f7, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:54:35,282 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/8a0a5cb16d414089b9f11d5750c6652d as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/8a0a5cb16d414089b9f11d5750c6652d 2023-10-27 23:54:35,286 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/8a0a5cb16d414089b9f11d5750c6652d, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:54:35,286 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for dfe214ec55570794a9b9dc2a65a69d99 in 33ms, sequenceid=6, compaction requested=false 2023-10-27 23:54:35,286 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionEmptyDaughter' 2023-10-27 23:54:35,287 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dfe214ec55570794a9b9dc2a65a69d99: 2023-10-27 23:54:35,287 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:35,287 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:35,287 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:35,287 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,287 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:35,287 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:35,296 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,296 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:35,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:35,297 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:35,297 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:35,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-10-27 23:54:35,298 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-10-27 23:54:35,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68d7caaf[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:35,298 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,307 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,307 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,307 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,308 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,308 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:35,308 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:35,308 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:35,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,308 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:35,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,308 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:35,308 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:35,308 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:35,308 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:35,308 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:35,308 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:35,316 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:35,316 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:35,316 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:35,324 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:35,324 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,324 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:35,324 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:35,324 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:35,324 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:35,325 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:54:35,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:54:35,327 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-10-27 23:54:35,327 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:35,327 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:35,327 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,327 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:35,338 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,338 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,338 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,338 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,338 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:35,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,338 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:35,338 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,338 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,339 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,339 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,340 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,341 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,341 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,385 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,385 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,385 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-10-27 23:54:35,385 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,385 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:35,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,385 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:35,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,385 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,385 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,385 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,385 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:35,385 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,386 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,386 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,386 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,386 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,386 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,386 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,386 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,386 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,386 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,486 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:35,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:35,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:35,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:35,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,500 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-10-27 23:54:35,500 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,500 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-10-27 23:54:35,500 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-10-27 23:54:35,500 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,500 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,516 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,516 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,516 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,516 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,516 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,516 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:35,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:35,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:35,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:35,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:35,518 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:35,518 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:35,518 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. started... 2023-10-27 23:54:35,518 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dfe214ec55570794a9b9dc2a65a69d99 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:35,524 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,524 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,524 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,524 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,524 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:35,524 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:35,526 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/d0d6f65d30104be996e434b62252799b 2023-10-27 23:54:35,535 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/790fa96d9d6c498088957c6510c38007 2023-10-27 23:54:35,538 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/d0d6f65d30104be996e434b62252799b as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/d0d6f65d30104be996e434b62252799b 2023-10-27 23:54:35,541 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/d0d6f65d30104be996e434b62252799b, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:54:35,542 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/790fa96d9d6c498088957c6510c38007 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/790fa96d9d6c498088957c6510c38007 2023-10-27 23:54:35,545 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/790fa96d9d6c498088957c6510c38007, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:54:35,545 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for dfe214ec55570794a9b9dc2a65a69d99 in 27ms, sequenceid=19, compaction requested=false 2023-10-27 23:54:35,546 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dfe214ec55570794a9b9dc2a65a69d99: 2023-10-27 23:54:35,546 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:35,546 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:35,546 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:35,546 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,546 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:35,546 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:35,554 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,554 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:35,555 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,555 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:35,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-10-27 23:54:35,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@737854df[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:35,557 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-10-27 23:54:35,557 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,566 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,566 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,566 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,566 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:35,566 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,566 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:35,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,566 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:35,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,566 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:35,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,567 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:35,567 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:35,567 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:35,567 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:35,567 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:35,567 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:35,574 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:35,574 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:35,574 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:35,582 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:35,582 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,582 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:35,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,583 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:35,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:35,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:35,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:35,591 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:35,591 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:35,591 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:35,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:35,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:54:35,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:54:35,594 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-10-27 23:54:35,594 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:35,594 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:35,594 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,594 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:35,604 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,604 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,604 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,604 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,605 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,605 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,605 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,605 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,605 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:35,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:35,606 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:35,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:35,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,607 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,607 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,607 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,616 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:35,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-10-27 23:54:35,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:35,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,616 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,617 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:35,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,617 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:35,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,617 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,617 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,617 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,617 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,617 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,617 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,617 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,717 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:35,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:35,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:35,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:35,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,729 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-10-27 23:54:35,729 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,729 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-10-27 23:54:35,730 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-10-27 23:54:35,730 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,730 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,738 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,738 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,738 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,738 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:35,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:35,739 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:35,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:35,739 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,739 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:35,739 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:35,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,740 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,740 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:35,740 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:35,740 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:35,740 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,740 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:35,740 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:35,740 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:35,740 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:35,740 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:35,740 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:35,740 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. started... 2023-10-27 23:54:35,740 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:35,740 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dfe214ec55570794a9b9dc2a65a69d99 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:35,749 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,753 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/4a232c1eab7943ae9b77223963625874 2023-10-27 23:54:35,769 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/39155260f23a4f20aa2a0b77b95d8862 2023-10-27 23:54:35,773 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/4a232c1eab7943ae9b77223963625874 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/4a232c1eab7943ae9b77223963625874 2023-10-27 23:54:35,777 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/4a232c1eab7943ae9b77223963625874, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:54:35,777 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/39155260f23a4f20aa2a0b77b95d8862 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/39155260f23a4f20aa2a0b77b95d8862 2023-10-27 23:54:35,778 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,778 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,778 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:35,778 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,778 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:35,780 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/39155260f23a4f20aa2a0b77b95d8862, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:54:35,781 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for dfe214ec55570794a9b9dc2a65a69d99 in 41ms, sequenceid=32, compaction requested=true 2023-10-27 23:54:35,781 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dfe214ec55570794a9b9dc2a65a69d99: 2023-10-27 23:54:35,781 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:35,781 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:35,781 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:35,781 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,781 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:35,781 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:35,791 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,791 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:35,791 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,791 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:35,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-10-27 23:54:35,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2de14580[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:35,793 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-10-27 23:54:35,793 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,799 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,799 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,799 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,799 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:35,799 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,800 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:35,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,799 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,800 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:35,800 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:35,800 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:35,800 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:35,800 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:35,800 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:35,800 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:35,807 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:35,807 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,808 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:35,808 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:35,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:35,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:35,816 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:35,816 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:35,816 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:35,816 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:35,816 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:35,816 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:54:35,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:54:35,819 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-10-27 23:54:35,819 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:35,819 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:35,819 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,819 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:35,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:35,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,825 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,825 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:35,825 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,825 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,827 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:35,827 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,827 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-10-27 23:54:35,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:35,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,838 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:35,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:35,838 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,838 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,839 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:35,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:35,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:35,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:35,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:35,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,839 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,939 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:35,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:35,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:35,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:35,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,952 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-10-27 23:54:35,952 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,952 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-10-27 23:54:35,952 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-10-27 23:54:35,953 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,953 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,963 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,963 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,963 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,963 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:35,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,963 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,963 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:35,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:35,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:35,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:35,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:35,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,965 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,965 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. started... 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:35,965 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dfe214ec55570794a9b9dc2a65a69d99 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:35,965 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:35,972 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/9da1a4c0cee248c9b23a92d6b0001efe 2023-10-27 23:54:35,974 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,974 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,974 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,974 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:35,974 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:35,974 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:35,982 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/f45774e20a314bc18029e1996be73a05 2023-10-27 23:54:35,986 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/9da1a4c0cee248c9b23a92d6b0001efe as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/9da1a4c0cee248c9b23a92d6b0001efe 2023-10-27 23:54:35,989 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/9da1a4c0cee248c9b23a92d6b0001efe, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:54:35,990 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/f45774e20a314bc18029e1996be73a05 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/f45774e20a314bc18029e1996be73a05 2023-10-27 23:54:35,993 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/f45774e20a314bc18029e1996be73a05, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:54:35,994 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for dfe214ec55570794a9b9dc2a65a69d99 in 29ms, sequenceid=45, compaction requested=true 2023-10-27 23:54:35,994 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dfe214ec55570794a9b9dc2a65a69d99: 2023-10-27 23:54:35,994 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:35,994 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:35,994 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:35,994 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:35,994 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:35,994 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:36,004 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,004 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:36,005 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:36,005 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:36,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-10-27 23:54:36,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@374a23fe[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:36,006 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-10-27 23:54:36,006 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,016 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,016 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,016 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,016 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,016 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:36,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,016 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:36,016 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:36,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,017 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:36,017 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:36,016 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:36,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,017 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:36,017 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:36,017 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:36,017 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:36,024 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:36,024 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:36,024 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:36,032 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:36,032 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,033 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:36,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:36,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:36,033 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:36,041 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:36,041 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:36,041 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:54:36,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:54:36,045 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-10-27 23:54:36,045 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:36,045 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:36,045 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,045 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:36,054 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,054 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,054 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,054 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,055 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:36,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,054 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,055 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,055 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:36,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,055 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,056 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:36,057 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,057 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,058 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,096 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,096 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,096 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,096 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,096 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,096 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,096 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,096 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-10-27 23:54:36,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:36,096 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:36,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,096 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,097 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:36,096 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,097 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:36,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,097 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,097 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,097 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,097 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,097 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,097 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,097 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,197 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:36,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:36,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:36,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:36,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,209 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-10-27 23:54:36,209 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:36,209 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-10-27 23:54:36,209 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-10-27 23:54:36,209 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,210 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,220 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,220 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,220 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,220 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,221 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,221 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:36,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:36,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:36,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:36,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,222 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:36,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,222 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:36,222 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:36,222 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:36,222 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:36,222 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:36,222 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:36,222 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:36,222 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:36,222 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:36,223 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:36,222 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:36,222 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:36,223 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:36,223 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:36,223 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:36,223 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. started... 2023-10-27 23:54:36,223 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dfe214ec55570794a9b9dc2a65a69d99 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:36,231 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/a353d3c80f4a444ca3288cf73a28ad17 2023-10-27 23:54:36,232 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,232 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,232 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,232 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,232 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:36,232 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:36,241 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/0d721705703644ea8069aff830d2fb1c 2023-10-27 23:54:36,244 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/a353d3c80f4a444ca3288cf73a28ad17 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/a353d3c80f4a444ca3288cf73a28ad17 2023-10-27 23:54:36,248 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/a353d3c80f4a444ca3288cf73a28ad17, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:54:36,248 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/0d721705703644ea8069aff830d2fb1c as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/0d721705703644ea8069aff830d2fb1c 2023-10-27 23:54:36,252 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/0d721705703644ea8069aff830d2fb1c, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:54:36,252 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for dfe214ec55570794a9b9dc2a65a69d99 in 29ms, sequenceid=58, compaction requested=true 2023-10-27 23:54:36,253 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dfe214ec55570794a9b9dc2a65a69d99: 2023-10-27 23:54:36,253 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:36,253 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:36,253 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:36,253 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,253 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:36,253 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:36,262 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,262 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:36,262 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:36,262 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:36,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-10-27 23:54:36,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611e050a[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:36,264 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-10-27 23:54:36,264 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,274 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,274 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,274 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,274 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,274 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,274 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,274 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:36,274 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:36,274 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:36,274 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:36,274 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:36,274 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:36,274 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:36,275 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:36,282 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:36,282 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:36,282 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:36,290 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:36,290 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,290 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:36,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:36,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:36,290 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:36,291 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:36,291 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:36,291 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:36,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:36,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:54:36,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:54:36,294 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-10-27 23:54:36,294 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:36,294 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:36,294 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,294 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:36,304 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,304 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,304 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,304 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,304 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,304 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,304 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,304 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:36,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,305 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,305 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,307 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,307 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:36,307 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,315 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,315 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,315 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,316 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:36,315 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-10-27 23:54:36,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:36,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,316 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,316 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:36,316 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,316 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:36,316 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,316 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,316 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,316 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,317 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,317 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,317 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,317 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,317 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,317 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,317 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,416 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:36,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:36,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:36,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:36,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,428 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-10-27 23:54:36,429 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:36,429 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-10-27 23:54:36,429 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-10-27 23:54:36,429 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,429 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,437 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,437 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,437 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,437 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,437 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,438 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:36,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:36,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:36,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:36,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:36,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:36,439 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:36,439 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:36,439 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. started... 2023-10-27 23:54:36,439 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dfe214ec55570794a9b9dc2a65a69d99 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:36,448 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/be7852c1195542ac9629764d60ee142d 2023-10-27 23:54:36,458 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/9dc9eb94e7074d25bffed3e4d30c879f 2023-10-27 23:54:36,461 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/be7852c1195542ac9629764d60ee142d as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/be7852c1195542ac9629764d60ee142d 2023-10-27 23:54:36,463 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,465 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/be7852c1195542ac9629764d60ee142d, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:54:36,465 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/9dc9eb94e7074d25bffed3e4d30c879f as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/9dc9eb94e7074d25bffed3e4d30c879f 2023-10-27 23:54:36,469 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/9dc9eb94e7074d25bffed3e4d30c879f, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:54:36,470 INFO [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for dfe214ec55570794a9b9dc2a65a69d99 in 31ms, sequenceid=71, compaction requested=true 2023-10-27 23:54:36,470 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dfe214ec55570794a9b9dc2a65a69d99: 2023-10-27 23:54:36,470 DEBUG [rs(jenkins-hbase16.apache.org,39059,1698450767147)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:36,470 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:36,470 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:36,470 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,470 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-10-27 23:54:36,470 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-10-27 23:54:36,474 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,474 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,474 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:36,482 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,482 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:36,482 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,482 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:36,483 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,483 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:36,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-10-27 23:54:36,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50717b5a[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:36,484 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-10-27 23:54:36,485 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,490 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,490 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,490 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,491 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:36,491 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:36,491 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:36,491 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,491 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:36,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,491 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:36,491 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:36,491 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:36,491 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-10-27 23:54:36,491 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-10-27 23:54:36,491 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:36,499 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:36,499 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,499 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:36,499 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:36,507 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:36,507 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-10-27 23:54:36,507 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:36,507 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:36,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:36,507 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:36,508 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-10-27 23:54:36,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:36,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:36,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase16.apache.org,39059,1698450767147': 2023-10-27 23:54:36,510 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-10-27 23:54:36,510 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:36,510 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:36,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-10-27 23:54:36,510 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,510 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:36,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,521 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:36,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,521 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,521 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,521 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,521 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,523 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:36,523 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,523 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,532 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,532 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,532 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,532 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:36,532 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:36,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-10-27 23:54:36,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:36,532 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,532 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,532 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,532 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,533 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:36,533 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,533 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:36,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:36,533 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:36,533 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:36,533 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,533 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,533 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:36,534 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:36,534 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,534 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-10-27 23:54:36,634 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:36,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:36,641 DEBUG [Listener at localhost/43373] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,642 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=dfe214ec55570794a9b9dc2a65a69d99, daughterA=eca6130fe4482ec1e18b9863b2d5298c, daughterB=1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:36,643 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=dfe214ec55570794a9b9dc2a65a69d99, daughterA=eca6130fe4482ec1e18b9863b2d5298c, daughterB=1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:36,643 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=dfe214ec55570794a9b9dc2a65a69d99, daughterA=eca6130fe4482ec1e18b9863b2d5298c, daughterB=1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:36,643 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=dfe214ec55570794a9b9dc2a65a69d99, daughterA=eca6130fe4482ec1e18b9863b2d5298c, daughterB=1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:36,645 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=129, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=dfe214ec55570794a9b9dc2a65a69d99, UNASSIGN}] 2023-10-27 23:54:36,646 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=129, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=dfe214ec55570794a9b9dc2a65a69d99, UNASSIGN 2023-10-27 23:54:36,646 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=129 updating hbase:meta row=dfe214ec55570794a9b9dc2a65a69d99, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,646 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450876646"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450876646"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450876646"}]},"ts":"1698450876646"} 2023-10-27 23:54:36,647 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=130, ppid=129, state=RUNNABLE; CloseRegionProcedure dfe214ec55570794a9b9dc2a65a69d99, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:54:36,686 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-27 23:54:36,798 DEBUG [RSProcedureDispatcher-pool-3] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:36,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,799 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing dfe214ec55570794a9b9dc2a65a69d99, disabling compactions & flushes 2023-10-27 23:54:36,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:36,799 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:36,799 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. after waiting 0 ms 2023-10-27 23:54:36,799 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:36,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing dfe214ec55570794a9b9dc2a65a69d99 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-10-27 23:54:36,807 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/c204adbf5eba40ac9e13645eb26cb17c 2023-10-27 23:54:36,818 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/5bd8e91f0e09487fbd74741fe5b2d69d 2023-10-27 23:54:36,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf1/c204adbf5eba40ac9e13645eb26cb17c as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/c204adbf5eba40ac9e13645eb26cb17c 2023-10-27 23:54:36,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/c204adbf5eba40ac9e13645eb26cb17c, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:54:36,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.tmp/cf2/5bd8e91f0e09487fbd74741fe5b2d69d as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/5bd8e91f0e09487fbd74741fe5b2d69d 2023-10-27 23:54:36,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/5bd8e91f0e09487fbd74741fe5b2d69d, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:54:36,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for dfe214ec55570794a9b9dc2a65a69d99 in 32ms, sequenceid=82, compaction requested=true 2023-10-27 23:54:36,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-10-27 23:54:36,835 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:36,835 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. 2023-10-27 23:54:36,835 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for dfe214ec55570794a9b9dc2a65a69d99: 2023-10-27 23:54:36,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,837 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=129 updating hbase:meta row=dfe214ec55570794a9b9dc2a65a69d99, regionState=CLOSED 2023-10-27 23:54:36,837 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450876837"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450876837"}]},"ts":"1698450876837"} 2023-10-27 23:54:36,839 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=130, resume processing ppid=129 2023-10-27 23:54:36,839 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=130, ppid=129, state=SUCCESS; CloseRegionProcedure dfe214ec55570794a9b9dc2a65a69d99, server=jenkins-hbase16.apache.org,39059,1698450767147 in 191 msec 2023-10-27 23:54:36,841 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=129, resume processing ppid=128 2023-10-27 23:54:36,841 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=129, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=dfe214ec55570794a9b9dc2a65a69d99, UNASSIGN in 194 msec 2023-10-27 23:54:36,850 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=128 splitting 14 storefiles, region=dfe214ec55570794a9b9dc2a65a69d99, threads=14 2023-10-27 23:54:36,850 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/0d721705703644ea8069aff830d2fb1c for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,850 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/39155260f23a4f20aa2a0b77b95d8862 for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,850 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/5bd8e91f0e09487fbd74741fe5b2d69d for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,850 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/790fa96d9d6c498088957c6510c38007 for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,851 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/8a0a5cb16d414089b9f11d5750c6652d for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,851 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/9dc9eb94e7074d25bffed3e4d30c879f for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,853 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/f45774e20a314bc18029e1996be73a05 for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,853 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/1e18611e034a4a77873e2c82048d05f7 for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,859 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/4a232c1eab7943ae9b77223963625874 for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,859 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/9da1a4c0cee248c9b23a92d6b0001efe for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,859 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/a353d3c80f4a444ca3288cf73a28ad17 for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,859 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/be7852c1195542ac9629764d60ee142d for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,859 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/c204adbf5eba40ac9e13645eb26cb17c for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,859 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/d0d6f65d30104be996e434b62252799b for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,863 DEBUG [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/5bd8e91f0e09487fbd74741fe5b2d69d, top=false 2023-10-27 23:54:36,864 DEBUG [StoreFileSplitter-pool-7] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/1e18611e034a4a77873e2c82048d05f7, top=false 2023-10-27 23:54:36,865 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/c204adbf5eba40ac9e13645eb26cb17c, top=false 2023-10-27 23:54:36,866 DEBUG [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/790fa96d9d6c498088957c6510c38007, top=false 2023-10-27 23:54:36,866 DEBUG [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/4a232c1eab7943ae9b77223963625874, top=false 2023-10-27 23:54:36,870 DEBUG [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/be7852c1195542ac9629764d60ee142d, top=false 2023-10-27 23:54:36,871 DEBUG [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/9da1a4c0cee248c9b23a92d6b0001efe, top=false 2023-10-27 23:54:36,871 DEBUG [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/9dc9eb94e7074d25bffed3e4d30c879f, top=false 2023-10-27 23:54:36,872 DEBUG [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/39155260f23a4f20aa2a0b77b95d8862, top=false 2023-10-27 23:54:36,873 DEBUG [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/8a0a5cb16d414089b9f11d5750c6652d, top=false 2023-10-27 23:54:36,873 DEBUG [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/0d721705703644ea8069aff830d2fb1c, top=false 2023-10-27 23:54:36,873 DEBUG [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/d0d6f65d30104be996e434b62252799b, top=false 2023-10-27 23:54:36,875 INFO [StoreFileSplitter-pool-7] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-1e18611e034a4a77873e2c82048d05f7 for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,878 DEBUG [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/a353d3c80f4a444ca3288cf73a28ad17, top=false 2023-10-27 23:54:36,878 DEBUG [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/f45774e20a314bc18029e1996be73a05, top=false 2023-10-27 23:54:36,879 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/1e18611e034a4a77873e2c82048d05f7 for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,887 INFO [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9dc9eb94e7074d25bffed3e4d30c879f for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,891 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/9dc9eb94e7074d25bffed3e4d30c879f for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,899 INFO [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-790fa96d9d6c498088957c6510c38007 for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,900 INFO [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-8a0a5cb16d414089b9f11d5750c6652d for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,900 INFO [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-be7852c1195542ac9629764d60ee142d for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,901 INFO [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-0d721705703644ea8069aff830d2fb1c for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,907 INFO [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-d0d6f65d30104be996e434b62252799b for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,908 INFO [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9da1a4c0cee248c9b23a92d6b0001efe for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,909 INFO [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-39155260f23a4f20aa2a0b77b95d8862 for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,909 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-c204adbf5eba40ac9e13645eb26cb17c for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,909 INFO [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-4a232c1eab7943ae9b77223963625874 for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,909 INFO [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-5bd8e91f0e09487fbd74741fe5b2d69d for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,910 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/790fa96d9d6c498088957c6510c38007 for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,912 INFO [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-f45774e20a314bc18029e1996be73a05 for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,912 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/8a0a5cb16d414089b9f11d5750c6652d for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,912 INFO [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/.splits/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-a353d3c80f4a444ca3288cf73a28ad17 for child: eca6130fe4482ec1e18b9863b2d5298c, parent: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,914 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/d0d6f65d30104be996e434b62252799b for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,914 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/c204adbf5eba40ac9e13645eb26cb17c for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,914 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/be7852c1195542ac9629764d60ee142d for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,914 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/9da1a4c0cee248c9b23a92d6b0001efe for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,915 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/0d721705703644ea8069aff830d2fb1c for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,916 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/39155260f23a4f20aa2a0b77b95d8862 for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,917 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/4a232c1eab7943ae9b77223963625874 for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,917 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/f45774e20a314bc18029e1996be73a05 for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,917 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/5bd8e91f0e09487fbd74741fe5b2d69d for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,917 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/a353d3c80f4a444ca3288cf73a28ad17 for region: dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:36,917 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=128 split storefiles for region dfe214ec55570794a9b9dc2a65a69d99 Daughter A: 14 storefiles, Daughter B: 0 storefiles. 2023-10-27 23:54:36,948 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-10-27 23:54:36,950 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-10-27 23:54:36,951 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1698450876951"},{"qualifier":"splitA","vlen":69,"tag":[],"timestamp":"1698450876951"},{"qualifier":"splitB","vlen":69,"tag":[],"timestamp":"1698450876951"}]},"ts":"1698450876951"} 2023-10-27 23:54:36,952 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1698450876951"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450876951"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450876951"}]},"ts":"1698450876951"} 2023-10-27 23:54:36,952 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1698450876951"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450876951"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450876951"}]},"ts":"1698450876951"} 2023-10-27 23:54:36,955 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eca6130fe4482ec1e18b9863b2d5298c, ASSIGN}, {pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=1effbe533d1888f1a96e39a217762cb6, ASSIGN}] 2023-10-27 23:54:36,956 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=1effbe533d1888f1a96e39a217762cb6, ASSIGN 2023-10-27 23:54:36,956 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=1effbe533d1888f1a96e39a217762cb6, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase16.apache.org,39059,1698450767147; forceNewPlan=false, retain=false 2023-10-27 23:54:36,957 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eca6130fe4482ec1e18b9863b2d5298c, ASSIGN 2023-10-27 23:54:36,957 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eca6130fe4482ec1e18b9863b2d5298c, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase16.apache.org,39059,1698450767147; forceNewPlan=false, retain=false 2023-10-27 23:54:37,107 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-10-27 23:54:37,108 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=132 updating hbase:meta row=1effbe533d1888f1a96e39a217762cb6, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:37,108 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1698450877108"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450877108"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450877108"}]},"ts":"1698450877108"} 2023-10-27 23:54:37,110 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=133, ppid=132, state=RUNNABLE; OpenRegionProcedure 1effbe533d1888f1a96e39a217762cb6, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:54:37,112 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=131 updating hbase:meta row=eca6130fe4482ec1e18b9863b2d5298c, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:37,112 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1698450877112"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450877112"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450877112"}]},"ts":"1698450877112"} 2023-10-27 23:54:37,113 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=134, ppid=131, state=RUNNABLE; OpenRegionProcedure eca6130fe4482ec1e18b9863b2d5298c, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:54:37,261 DEBUG [RSProcedureDispatcher-pool-4] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:37,264 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c. 2023-10-27 23:54:37,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => eca6130fe4482ec1e18b9863b2d5298c, NAME => 'testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.', STARTKEY => '', ENDKEY => '71'} 2023-10-27 23:54:37,264 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:54:37,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter eca6130fe4482ec1e18b9863b2d5298c 2023-10-27 23:54:37,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:37,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for eca6130fe4482ec1e18b9863b2d5298c 2023-10-27 23:54:37,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for eca6130fe4482ec1e18b9863b2d5298c 2023-10-27 23:54:37,266 INFO [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region eca6130fe4482ec1e18b9863b2d5298c 2023-10-27 23:54:37,266 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1 2023-10-27 23:54:37,267 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1 2023-10-27 23:54:37,267 INFO [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eca6130fe4482ec1e18b9863b2d5298c columnFamilyName cf1 2023-10-27 23:54:37,273 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-1e18611e034a4a77873e2c82048d05f7 2023-10-27 23:54:37,276 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-4a232c1eab7943ae9b77223963625874 2023-10-27 23:54:37,279 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9da1a4c0cee248c9b23a92d6b0001efe 2023-10-27 23:54:37,281 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-a353d3c80f4a444ca3288cf73a28ad17 2023-10-27 23:54:37,284 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-be7852c1195542ac9629764d60ee142d 2023-10-27 23:54:37,287 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-c204adbf5eba40ac9e13645eb26cb17c 2023-10-27 23:54:37,289 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-d0d6f65d30104be996e434b62252799b 2023-10-27 23:54:37,290 INFO [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(310): Store=eca6130fe4482ec1e18b9863b2d5298c/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:37,290 INFO [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region eca6130fe4482ec1e18b9863b2d5298c 2023-10-27 23:54:37,290 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2 2023-10-27 23:54:37,290 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2 2023-10-27 23:54:37,291 INFO [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eca6130fe4482ec1e18b9863b2d5298c columnFamilyName cf2 2023-10-27 23:54:37,295 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-0d721705703644ea8069aff830d2fb1c 2023-10-27 23:54:37,297 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-39155260f23a4f20aa2a0b77b95d8862 2023-10-27 23:54:37,300 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-5bd8e91f0e09487fbd74741fe5b2d69d 2023-10-27 23:54:37,303 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-790fa96d9d6c498088957c6510c38007 2023-10-27 23:54:37,305 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-8a0a5cb16d414089b9f11d5750c6652d 2023-10-27 23:54:37,308 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9dc9eb94e7074d25bffed3e4d30c879f 2023-10-27 23:54:37,311 DEBUG [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-f45774e20a314bc18029e1996be73a05 2023-10-27 23:54:37,311 INFO [StoreOpener-eca6130fe4482ec1e18b9863b2d5298c-1] regionserver.HStore(310): Store=eca6130fe4482ec1e18b9863b2d5298c/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:37,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c 2023-10-27 23:54:37,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c 2023-10-27 23:54:37,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:54:37,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for eca6130fe4482ec1e18b9863b2d5298c 2023-10-27 23:54:37,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened eca6130fe4482ec1e18b9863b2d5298c; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68573388, jitterRate=0.021823108196258545}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:54:37,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for eca6130fe4482ec1e18b9863b2d5298c: 2023-10-27 23:54:37,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c., pid=134, masterSystemTime=1698450877261 2023-10-27 23:54:37,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:37,316 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:54:37,316 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:54:37,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:54:37,317 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-10-27 23:54:37,317 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-10-27 23:54:37,317 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1912): eca6130fe4482ec1e18b9863b2d5298c/cf1 is initiating minor compaction (all files) 2023-10-27 23:54:37,317 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1912): eca6130fe4482ec1e18b9863b2d5298c/cf2 is initiating minor compaction (all files) 2023-10-27 23:54:37,317 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of eca6130fe4482ec1e18b9863b2d5298c/cf1 in testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c. 2023-10-27 23:54:37,317 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegion(2259): Starting compaction of eca6130fe4482ec1e18b9863b2d5298c/cf2 in testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c. 2023-10-27 23:54:37,317 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-8a0a5cb16d414089b9f11d5750c6652d, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-790fa96d9d6c498088957c6510c38007, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-39155260f23a4f20aa2a0b77b95d8862, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-f45774e20a314bc18029e1996be73a05, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-0d721705703644ea8069aff830d2fb1c, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9dc9eb94e7074d25bffed3e4d30c879f, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-5bd8e91f0e09487fbd74741fe5b2d69d] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/.tmp, totalSize=34.9 K 2023-10-27 23:54:37,317 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-1e18611e034a4a77873e2c82048d05f7, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-d0d6f65d30104be996e434b62252799b, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-4a232c1eab7943ae9b77223963625874, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9da1a4c0cee248c9b23a92d6b0001efe, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-a353d3c80f4a444ca3288cf73a28ad17, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-be7852c1195542ac9629764d60ee142d, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-c204adbf5eba40ac9e13645eb26cb17c] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/.tmp, totalSize=34.9 K 2023-10-27 23:54:37,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c. 2023-10-27 23:54:37,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c. 2023-10-27 23:54:37,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6. 2023-10-27 23:54:37,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1effbe533d1888f1a96e39a217762cb6, NAME => 'testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6.', STARTKEY => '71', ENDKEY => ''} 2023-10-27 23:54:37,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:54:37,318 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-8a0a5cb16d414089b9f11d5750c6652d, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450875234 2023-10-27 23:54:37,318 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-1e18611e034a4a77873e2c82048d05f7, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450875234 2023-10-27 23:54:37,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter 1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:37,318 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=131 updating hbase:meta row=eca6130fe4482ec1e18b9863b2d5298c, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:37,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:37,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:37,319 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1698450877318"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450877318"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450877318"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450877318"}]},"ts":"1698450877318"} 2023-10-27 23:54:37,319 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-790fa96d9d6c498088957c6510c38007, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450875488 2023-10-27 23:54:37,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:37,319 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-d0d6f65d30104be996e434b62252799b, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450875488 2023-10-27 23:54:37,319 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-4a232c1eab7943ae9b77223963625874, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450875719 2023-10-27 23:54:37,319 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-39155260f23a4f20aa2a0b77b95d8862, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450875719 2023-10-27 23:54:37,320 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-f45774e20a314bc18029e1996be73a05, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450875941 2023-10-27 23:54:37,320 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9da1a4c0cee248c9b23a92d6b0001efe, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450875941 2023-10-27 23:54:37,320 INFO [StoreOpener-1effbe533d1888f1a96e39a217762cb6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:37,320 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-0d721705703644ea8069aff830d2fb1c, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450876199 2023-10-27 23:54:37,320 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-a353d3c80f4a444ca3288cf73a28ad17, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450876199 2023-10-27 23:54:37,321 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-be7852c1195542ac9629764d60ee142d, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450876418 2023-10-27 23:54:37,321 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9dc9eb94e7074d25bffed3e4d30c879f, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450876418 2023-10-27 23:54:37,321 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-c204adbf5eba40ac9e13645eb26cb17c, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1698450876635 2023-10-27 23:54:37,322 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=134, resume processing ppid=131 2023-10-27 23:54:37,322 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-5bd8e91f0e09487fbd74741fe5b2d69d, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1698450876635 2023-10-27 23:54:37,322 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=134, ppid=131, state=SUCCESS; OpenRegionProcedure eca6130fe4482ec1e18b9863b2d5298c, server=jenkins-hbase16.apache.org,39059,1698450767147 in 207 msec 2023-10-27 23:54:37,322 DEBUG [StoreOpener-1effbe533d1888f1a96e39a217762cb6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6/cf1 2023-10-27 23:54:37,322 DEBUG [StoreOpener-1effbe533d1888f1a96e39a217762cb6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6/cf1 2023-10-27 23:54:37,322 INFO [StoreOpener-1effbe533d1888f1a96e39a217762cb6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1effbe533d1888f1a96e39a217762cb6 columnFamilyName cf1 2023-10-27 23:54:37,323 INFO [StoreOpener-1effbe533d1888f1a96e39a217762cb6-1] regionserver.HStore(310): Store=1effbe533d1888f1a96e39a217762cb6/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:37,323 INFO [StoreOpener-1effbe533d1888f1a96e39a217762cb6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:37,324 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=131, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eca6130fe4482ec1e18b9863b2d5298c, ASSIGN in 367 msec 2023-10-27 23:54:37,324 DEBUG [StoreOpener-1effbe533d1888f1a96e39a217762cb6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6/cf2 2023-10-27 23:54:37,325 DEBUG [StoreOpener-1effbe533d1888f1a96e39a217762cb6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6/cf2 2023-10-27 23:54:37,325 INFO [StoreOpener-1effbe533d1888f1a96e39a217762cb6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1effbe533d1888f1a96e39a217762cb6 columnFamilyName cf2 2023-10-27 23:54:37,325 INFO [StoreOpener-1effbe533d1888f1a96e39a217762cb6-1] regionserver.HStore(310): Store=1effbe533d1888f1a96e39a217762cb6/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:37,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:37,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:37,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:54:37,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:37,332 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 1effbe533d1888f1a96e39a217762cb6; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69439935, jitterRate=0.03473566472530365}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:54:37,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 1effbe533d1888f1a96e39a217762cb6: 2023-10-27 23:54:37,332 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6., pid=133, masterSystemTime=1698450877261 2023-10-27 23:54:37,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6. 2023-10-27 23:54:37,334 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6. 2023-10-27 23:54:37,334 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=132 updating hbase:meta row=1effbe533d1888f1a96e39a217762cb6, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:37,335 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1698450877334"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450877334"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450877334"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450877334"}]},"ts":"1698450877334"} 2023-10-27 23:54:37,337 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=133, resume processing ppid=132 2023-10-27 23:54:37,337 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=133, ppid=132, state=SUCCESS; OpenRegionProcedure 1effbe533d1888f1a96e39a217762cb6, server=jenkins-hbase16.apache.org,39059,1698450767147 in 226 msec 2023-10-27 23:54:37,339 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=132, resume processing ppid=128 2023-10-27 23:54:37,339 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=132, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=1effbe533d1888f1a96e39a217762cb6, ASSIGN in 382 msec 2023-10-27 23:54:37,341 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=128, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=dfe214ec55570794a9b9dc2a65a69d99, daughterA=eca6130fe4482ec1e18b9863b2d5298c, daughterB=1effbe533d1888f1a96e39a217762cb6 in 697 msec 2023-10-27 23:54:37,344 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] throttle.PressureAwareThroughputController(145): eca6130fe4482ec1e18b9863b2d5298c#cf2#compaction#126 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:54:37,348 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] throttle.PressureAwareThroughputController(145): eca6130fe4482ec1e18b9863b2d5298c#cf1#compaction#127 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:54:37,363 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/.tmp/cf2/33ea3e7f79ed4326b0b2bf31907b8f7e as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/33ea3e7f79ed4326b0b2bf31907b8f7e 2023-10-27 23:54:37,364 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/.tmp/cf1/8acc88aeab994ba287f9505a790b4fe2 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/8acc88aeab994ba287f9505a790b4fe2 2023-10-27 23:54:37,367 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in eca6130fe4482ec1e18b9863b2d5298c/cf2 of eca6130fe4482ec1e18b9863b2d5298c into 33ea3e7f79ed4326b0b2bf31907b8f7e(size=7.3 K), total size for store is 7.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:54:37,367 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for eca6130fe4482ec1e18b9863b2d5298c: 2023-10-27 23:54:37,367 INFO [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c., storeName=eca6130fe4482ec1e18b9863b2d5298c/cf2, priority=9, startTime=1698450877316; duration=0sec 2023-10-27 23:54:37,368 DEBUG [RS:0;jenkins-hbase16:39059-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:37,369 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in eca6130fe4482ec1e18b9863b2d5298c/cf1 of eca6130fe4482ec1e18b9863b2d5298c into 8acc88aeab994ba287f9505a790b4fe2(size=7.3 K), total size for store is 7.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:54:37,370 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for eca6130fe4482ec1e18b9863b2d5298c: 2023-10-27 23:54:37,370 INFO [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c., storeName=eca6130fe4482ec1e18b9863b2d5298c/cf1, priority=9, startTime=1698450877316; duration=0sec 2023-10-27 23:54:37,370 DEBUG [RS:0;jenkins-hbase16:39059-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:37,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39059] ipc.CallRunner(144): callId: 999 service: ClientService methodName: Scan size: 157 connection: 188.40.62.62:58066 deadline: 1698450937393, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99. is not online on jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:37,504 DEBUG [Listener at localhost/43373] client.ClientScanner(240): Advancing internal scanner to startKey at '71', inclusive 2023-10-27 23:54:37,507 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:54:37,508 INFO [Listener at localhost/43373] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionEmptyDaughter 2023-10-27 23:54:37,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable testSplitTableRegionEmptyDaughter 2023-10-27 23:54:37,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=135, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionEmptyDaughter 2023-10-27 23:54:37,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=135 2023-10-27 23:54:37,511 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450877511"}]},"ts":"1698450877511"} 2023-10-27 23:54:37,512 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=DISABLING in hbase:meta 2023-10-27 23:54:37,545 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionEmptyDaughter to state=DISABLING 2023-10-27 23:54:37,546 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=136, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eca6130fe4482ec1e18b9863b2d5298c, UNASSIGN}, {pid=137, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=1effbe533d1888f1a96e39a217762cb6, UNASSIGN}] 2023-10-27 23:54:37,547 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=1effbe533d1888f1a96e39a217762cb6, UNASSIGN 2023-10-27 23:54:37,547 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=1effbe533d1888f1a96e39a217762cb6, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:37,547 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1698450877547"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450877547"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450877547"}]},"ts":"1698450877547"} 2023-10-27 23:54:37,548 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=138, ppid=137, state=RUNNABLE; CloseRegionProcedure 1effbe533d1888f1a96e39a217762cb6, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:54:37,550 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eca6130fe4482ec1e18b9863b2d5298c, UNASSIGN 2023-10-27 23:54:37,551 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=eca6130fe4482ec1e18b9863b2d5298c, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:37,551 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1698450877551"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450877551"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450877551"}]},"ts":"1698450877551"} 2023-10-27 23:54:37,552 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=139, ppid=136, state=RUNNABLE; CloseRegionProcedure eca6130fe4482ec1e18b9863b2d5298c, server=jenkins-hbase16.apache.org,39059,1698450767147}] 2023-10-27 23:54:37,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=135 2023-10-27 23:54:37,700 DEBUG [RSProcedureDispatcher-pool-5] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:37,700 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:37,701 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 1effbe533d1888f1a96e39a217762cb6, disabling compactions & flushes 2023-10-27 23:54:37,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6. 2023-10-27 23:54:37,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6. 2023-10-27 23:54:37,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6. after waiting 0 ms 2023-10-27 23:54:37,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6. 2023-10-27 23:54:37,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6/recovered.edits/88.seqid, newMaxSeqId=88, maxSeqId=85 2023-10-27 23:54:37,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:37,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6. 2023-10-27 23:54:37,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 1effbe533d1888f1a96e39a217762cb6: 2023-10-27 23:54:37,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:37,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close eca6130fe4482ec1e18b9863b2d5298c 2023-10-27 23:54:37,707 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing eca6130fe4482ec1e18b9863b2d5298c, disabling compactions & flushes 2023-10-27 23:54:37,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c. 2023-10-27 23:54:37,707 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c. 2023-10-27 23:54:37,707 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c. after waiting 0 ms 2023-10-27 23:54:37,707 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c. 2023-10-27 23:54:37,707 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=1effbe533d1888f1a96e39a217762cb6, regionState=CLOSED 2023-10-27 23:54:37,707 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1698450877707"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450877707"}]},"ts":"1698450877707"} 2023-10-27 23:54:37,708 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-1e18611e034a4a77873e2c82048d05f7, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-d0d6f65d30104be996e434b62252799b, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-4a232c1eab7943ae9b77223963625874, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9da1a4c0cee248c9b23a92d6b0001efe, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-a353d3c80f4a444ca3288cf73a28ad17, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-be7852c1195542ac9629764d60ee142d, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-c204adbf5eba40ac9e13645eb26cb17c] to archive 2023-10-27 23:54:37,708 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:54:37,710 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=138, resume processing ppid=137 2023-10-27 23:54:37,710 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=138, ppid=137, state=SUCCESS; CloseRegionProcedure 1effbe533d1888f1a96e39a217762cb6, server=jenkins-hbase16.apache.org,39059,1698450767147 in 160 msec 2023-10-27 23:54:37,710 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-1e18611e034a4a77873e2c82048d05f7 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-1e18611e034a4a77873e2c82048d05f7 2023-10-27 23:54:37,711 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-d0d6f65d30104be996e434b62252799b to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-d0d6f65d30104be996e434b62252799b 2023-10-27 23:54:37,711 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=137, ppid=135, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=1effbe533d1888f1a96e39a217762cb6, UNASSIGN in 164 msec 2023-10-27 23:54:37,712 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-4a232c1eab7943ae9b77223963625874 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-4a232c1eab7943ae9b77223963625874 2023-10-27 23:54:37,713 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9da1a4c0cee248c9b23a92d6b0001efe to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9da1a4c0cee248c9b23a92d6b0001efe 2023-10-27 23:54:37,714 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-a353d3c80f4a444ca3288cf73a28ad17 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-a353d3c80f4a444ca3288cf73a28ad17 2023-10-27 23:54:37,715 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-be7852c1195542ac9629764d60ee142d to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-be7852c1195542ac9629764d60ee142d 2023-10-27 23:54:37,715 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-c204adbf5eba40ac9e13645eb26cb17c to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-c204adbf5eba40ac9e13645eb26cb17c 2023-10-27 23:54:37,717 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-8a0a5cb16d414089b9f11d5750c6652d, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-790fa96d9d6c498088957c6510c38007, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-39155260f23a4f20aa2a0b77b95d8862, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-f45774e20a314bc18029e1996be73a05, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-0d721705703644ea8069aff830d2fb1c, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9dc9eb94e7074d25bffed3e4d30c879f, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-5bd8e91f0e09487fbd74741fe5b2d69d] to archive 2023-10-27 23:54:37,717 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:54:37,718 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-8a0a5cb16d414089b9f11d5750c6652d to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-8a0a5cb16d414089b9f11d5750c6652d 2023-10-27 23:54:37,719 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-790fa96d9d6c498088957c6510c38007 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-790fa96d9d6c498088957c6510c38007 2023-10-27 23:54:37,720 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-39155260f23a4f20aa2a0b77b95d8862 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-39155260f23a4f20aa2a0b77b95d8862 2023-10-27 23:54:37,721 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-f45774e20a314bc18029e1996be73a05 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-f45774e20a314bc18029e1996be73a05 2023-10-27 23:54:37,721 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-0d721705703644ea8069aff830d2fb1c to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-0d721705703644ea8069aff830d2fb1c 2023-10-27 23:54:37,722 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9dc9eb94e7074d25bffed3e4d30c879f to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-9dc9eb94e7074d25bffed3e4d30c879f 2023-10-27 23:54:37,723 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-5bd8e91f0e09487fbd74741fe5b2d69d to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/testSplitTableRegionEmptyDaughter=dfe214ec55570794a9b9dc2a65a69d99-5bd8e91f0e09487fbd74741fe5b2d69d 2023-10-27 23:54:37,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-10-27 23:54:37,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:37,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c. 2023-10-27 23:54:37,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for eca6130fe4482ec1e18b9863b2d5298c: 2023-10-27 23:54:37,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed eca6130fe4482ec1e18b9863b2d5298c 2023-10-27 23:54:37,728 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=eca6130fe4482ec1e18b9863b2d5298c, regionState=CLOSED 2023-10-27 23:54:37,728 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1698450877728"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450877728"}]},"ts":"1698450877728"} 2023-10-27 23:54:37,730 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=139, resume processing ppid=136 2023-10-27 23:54:37,730 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=139, ppid=136, state=SUCCESS; CloseRegionProcedure eca6130fe4482ec1e18b9863b2d5298c, server=jenkins-hbase16.apache.org,39059,1698450767147 in 177 msec 2023-10-27 23:54:37,732 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=136, resume processing ppid=135 2023-10-27 23:54:37,732 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=136, ppid=135, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eca6130fe4482ec1e18b9863b2d5298c, UNASSIGN in 184 msec 2023-10-27 23:54:37,732 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450877732"}]},"ts":"1698450877732"} 2023-10-27 23:54:37,733 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=DISABLED in hbase:meta 2023-10-27 23:54:37,740 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionEmptyDaughter to state=DISABLED 2023-10-27 23:54:37,742 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=135, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionEmptyDaughter in 233 msec 2023-10-27 23:54:37,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=135 2023-10-27 23:54:37,814 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionEmptyDaughter, procId: 135 completed 2023-10-27 23:54:37,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete testSplitTableRegionEmptyDaughter 2023-10-27 23:54:37,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=140, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-10-27 23:54:37,817 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=140, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-10-27 23:54:37,818 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=140, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-10-27 23:54:37,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=140 2023-10-27 23:54:37,822 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:37,822 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:37,822 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c 2023-10-27 23:54:37,823 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6/recovered.edits] 2023-10-27 23:54:37,824 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/recovered.edits] 2023-10-27 23:54:37,824 DEBUG [HFileArchiver-10] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/recovered.edits] 2023-10-27 23:54:37,826 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/1e18611e034a4a77873e2c82048d05f7 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/1e18611e034a4a77873e2c82048d05f7 2023-10-27 23:54:37,826 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/8acc88aeab994ba287f9505a790b4fe2 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf1/8acc88aeab994ba287f9505a790b4fe2 2023-10-27 23:54:37,827 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/4a232c1eab7943ae9b77223963625874 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/4a232c1eab7943ae9b77223963625874 2023-10-27 23:54:37,828 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6/recovered.edits/88.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6/recovered.edits/88.seqid 2023-10-27 23:54:37,828 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/33ea3e7f79ed4326b0b2bf31907b8f7e to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/cf2/33ea3e7f79ed4326b0b2bf31907b8f7e 2023-10-27 23:54:37,828 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/9da1a4c0cee248c9b23a92d6b0001efe to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/9da1a4c0cee248c9b23a92d6b0001efe 2023-10-27 23:54:37,828 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/1effbe533d1888f1a96e39a217762cb6 2023-10-27 23:54:37,829 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/a353d3c80f4a444ca3288cf73a28ad17 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/a353d3c80f4a444ca3288cf73a28ad17 2023-10-27 23:54:37,830 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/be7852c1195542ac9629764d60ee142d to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/be7852c1195542ac9629764d60ee142d 2023-10-27 23:54:37,830 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/recovered.edits/90.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c/recovered.edits/90.seqid 2023-10-27 23:54:37,830 DEBUG [HFileArchiver-10] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/eca6130fe4482ec1e18b9863b2d5298c 2023-10-27 23:54:37,831 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/c204adbf5eba40ac9e13645eb26cb17c to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/c204adbf5eba40ac9e13645eb26cb17c 2023-10-27 23:54:37,832 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/d0d6f65d30104be996e434b62252799b to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf1/d0d6f65d30104be996e434b62252799b 2023-10-27 23:54:37,833 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/0d721705703644ea8069aff830d2fb1c to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/0d721705703644ea8069aff830d2fb1c 2023-10-27 23:54:37,834 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/39155260f23a4f20aa2a0b77b95d8862 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/39155260f23a4f20aa2a0b77b95d8862 2023-10-27 23:54:37,835 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/5bd8e91f0e09487fbd74741fe5b2d69d to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/5bd8e91f0e09487fbd74741fe5b2d69d 2023-10-27 23:54:37,836 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/790fa96d9d6c498088957c6510c38007 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/790fa96d9d6c498088957c6510c38007 2023-10-27 23:54:37,836 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/8a0a5cb16d414089b9f11d5750c6652d to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/8a0a5cb16d414089b9f11d5750c6652d 2023-10-27 23:54:37,837 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/9dc9eb94e7074d25bffed3e4d30c879f to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/9dc9eb94e7074d25bffed3e4d30c879f 2023-10-27 23:54:37,838 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/f45774e20a314bc18029e1996be73a05 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/cf2/f45774e20a314bc18029e1996be73a05 2023-10-27 23:54:37,840 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/recovered.edits/85.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99/recovered.edits/85.seqid 2023-10-27 23:54:37,840 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegionEmptyDaughter/dfe214ec55570794a9b9dc2a65a69d99 2023-10-27 23:54:37,840 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionEmptyDaughter regions 2023-10-27 23:54:37,842 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=140, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-10-27 23:54:37,844 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionEmptyDaughter from hbase:meta 2023-10-27 23:54:37,845 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionEmptyDaughter' descriptor. 2023-10-27 23:54:37,846 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=140, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-10-27 23:54:37,846 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionEmptyDaughter' from region states. 2023-10-27 23:54:37,846 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450877846"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:37,847 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450877846"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:37,847 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450877846"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:37,848 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-10-27 23:54:37,848 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => dfe214ec55570794a9b9dc2a65a69d99, NAME => 'testSplitTableRegionEmptyDaughter,,1698450874730.dfe214ec55570794a9b9dc2a65a69d99.', STARTKEY => '', ENDKEY => ''}, {ENCODED => eca6130fe4482ec1e18b9863b2d5298c, NAME => 'testSplitTableRegionEmptyDaughter,,1698450876642.eca6130fe4482ec1e18b9863b2d5298c.', STARTKEY => '', ENDKEY => '71'}, {ENCODED => 1effbe533d1888f1a96e39a217762cb6, NAME => 'testSplitTableRegionEmptyDaughter,71,1698450876642.1effbe533d1888f1a96e39a217762cb6.', STARTKEY => '71', ENDKEY => ''}] 2023-10-27 23:54:37,848 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionEmptyDaughter' as deleted. 2023-10-27 23:54:37,848 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1698450877848"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:37,850 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionEmptyDaughter state from META 2023-10-27 23:54:37,858 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=140, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-10-27 23:54:37,859 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=140, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter in 43 msec 2023-10-27 23:54:37,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=140 2023-10-27 23:54:37,919 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionEmptyDaughter, procId: 140 completed 2023-10-27 23:54:37,930 INFO [Listener at localhost/43373] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionEmptyDaughter Thread=379 (was 379), OpenFileDescriptor=841 (was 841), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=214 (was 224), ProcessCount=170 (was 170), AvailableMemoryMB=5396 (was 5402) 2023-10-27 23:54:37,942 INFO [Listener at localhost/43373] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegion Thread=379, OpenFileDescriptor=841, MaxFileDescriptor=60000, SystemLoadAverage=214, ProcessCount=170, AvailableMemoryMB=5395 2023-10-27 23:54:37,942 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:54:37,954 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-10-27 23:54:37,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(492): Client=jenkins//188.40.62.62 set balanceSwitch=false 2023-10-27 23:54:37,956 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=141, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegion 2023-10-27 23:54:37,957 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_PRE_OPERATION 2023-10-27 23:54:37,958 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-10-27 23:54:37,959 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:37,959 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf empty. 2023-10-27 23:54:37,960 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:37,960 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegion regions 2023-10-27 23:54:37,968 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/.tabledesc/.tableinfo.0000000001 2023-10-27 23:54:37,969 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(7675): creating {ENCODED => a9cee4e7411fb3bdf7d42508bea93aaf, NAME => 'testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegion', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp 2023-10-27 23:54:37,975 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:37,975 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1604): Closing a9cee4e7411fb3bdf7d42508bea93aaf, disabling compactions & flushes 2023-10-27 23:54:37,975 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:37,975 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:37,975 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. after waiting 0 ms 2023-10-27 23:54:37,975 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:37,975 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:37,975 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1558): Region close journal for a9cee4e7411fb3bdf7d42508bea93aaf: 2023-10-27 23:54:37,977 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_ADD_TO_META 2023-10-27 23:54:37,978 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450877978"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450877978"}]},"ts":"1698450877978"} 2023-10-27 23:54:37,979 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-10-27 23:54:37,980 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-10-27 23:54:37,980 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450877980"}]},"ts":"1698450877980"} 2023-10-27 23:54:37,981 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=ENABLING in hbase:meta 2023-10-27 23:54:37,999 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-10-27 23:54:38,000 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-10-27 23:54:38,000 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-10-27 23:54:38,000 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-10-27 23:54:38,000 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-10-27 23:54:38,000 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=a9cee4e7411fb3bdf7d42508bea93aaf, ASSIGN}] 2023-10-27 23:54:38,002 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=a9cee4e7411fb3bdf7d42508bea93aaf, ASSIGN 2023-10-27 23:54:38,002 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=a9cee4e7411fb3bdf7d42508bea93aaf, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,43641,1698450767195; forceNewPlan=false, retain=false 2023-10-27 23:54:38,069 ERROR [BP-446492043-188.40.62.62-1698450761445 heartbeating to localhost/127.0.0.1:46525] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-446492043-188.40.62.62-1698450761445 (Datanode Uuid 9d32eec7-ccca-436f-a2e6-4184d95cd961) service to localhost/127.0.0.1:46525 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-10-27 23:54:38,070 ERROR [BP-446492043-188.40.62.62-1698450761445 heartbeating to localhost/127.0.0.1:46525] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-446492043-188.40.62.62-1698450761445 (Datanode Uuid 41efbad2-7a69-4304-ba15-0342fa6d22c3) service to localhost/127.0.0.1:46525 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-10-27 23:54:38,152 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-10-27 23:54:38,153 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=142 updating hbase:meta row=a9cee4e7411fb3bdf7d42508bea93aaf, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:38,154 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450878153"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450878153"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450878153"}]},"ts":"1698450878153"} 2023-10-27 23:54:38,155 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=143, ppid=142, state=RUNNABLE; OpenRegionProcedure a9cee4e7411fb3bdf7d42508bea93aaf, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:54:38,306 DEBUG [RSProcedureDispatcher-pool-3] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:38,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:38,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a9cee4e7411fb3bdf7d42508bea93aaf, NAME => 'testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf.', STARTKEY => '', ENDKEY => ''} 2023-10-27 23:54:38,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:54:38,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:38,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:38,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:38,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:38,311 INFO [StoreOpener-a9cee4e7411fb3bdf7d42508bea93aaf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:38,312 DEBUG [StoreOpener-a9cee4e7411fb3bdf7d42508bea93aaf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1 2023-10-27 23:54:38,312 DEBUG [StoreOpener-a9cee4e7411fb3bdf7d42508bea93aaf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1 2023-10-27 23:54:38,313 INFO [StoreOpener-a9cee4e7411fb3bdf7d42508bea93aaf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9cee4e7411fb3bdf7d42508bea93aaf columnFamilyName cf1 2023-10-27 23:54:38,313 INFO [StoreOpener-a9cee4e7411fb3bdf7d42508bea93aaf-1] regionserver.HStore(310): Store=a9cee4e7411fb3bdf7d42508bea93aaf/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:38,313 INFO [StoreOpener-a9cee4e7411fb3bdf7d42508bea93aaf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:38,314 DEBUG [StoreOpener-a9cee4e7411fb3bdf7d42508bea93aaf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2 2023-10-27 23:54:38,314 DEBUG [StoreOpener-a9cee4e7411fb3bdf7d42508bea93aaf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2 2023-10-27 23:54:38,315 INFO [StoreOpener-a9cee4e7411fb3bdf7d42508bea93aaf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9cee4e7411fb3bdf7d42508bea93aaf columnFamilyName cf2 2023-10-27 23:54:38,315 INFO [StoreOpener-a9cee4e7411fb3bdf7d42508bea93aaf-1] regionserver.HStore(310): Store=a9cee4e7411fb3bdf7d42508bea93aaf/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:38,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:38,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:38,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:54:38,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:38,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-10-27 23:54:38,320 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened a9cee4e7411fb3bdf7d42508bea93aaf; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60054792, jitterRate=-0.10511386394500732}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:54:38,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for a9cee4e7411fb3bdf7d42508bea93aaf: 2023-10-27 23:54:38,321 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf., pid=143, masterSystemTime=1698450878306 2023-10-27 23:54:38,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:38,322 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:38,322 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=142 updating hbase:meta row=a9cee4e7411fb3bdf7d42508bea93aaf, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:38,322 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450878322"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450878322"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450878322"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450878322"}]},"ts":"1698450878322"} 2023-10-27 23:54:38,324 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=143, resume processing ppid=142 2023-10-27 23:54:38,324 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=143, ppid=142, state=SUCCESS; OpenRegionProcedure a9cee4e7411fb3bdf7d42508bea93aaf, server=jenkins-hbase16.apache.org,43641,1698450767195 in 168 msec 2023-10-27 23:54:38,325 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=142, resume processing ppid=141 2023-10-27 23:54:38,325 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=142, ppid=141, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=a9cee4e7411fb3bdf7d42508bea93aaf, ASSIGN in 324 msec 2023-10-27 23:54:38,326 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-10-27 23:54:38,326 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450878326"}]},"ts":"1698450878326"} 2023-10-27 23:54:38,327 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=ENABLED in hbase:meta 2023-10-27 23:54:38,358 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_POST_OPERATION 2023-10-27 23:54:38,359 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=141, state=SUCCESS; CreateTableProcedure table=testSplitTableRegion in 403 msec 2023-10-27 23:54:38,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:38,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-10-27 23:54:38,463 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-10-27 23:54:38,463 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:38,463 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-10-27 23:54:38,463 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-10-27 23:54:38,464 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:38,464 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:38,474 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:38,474 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:38,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:38,474 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:38,474 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:38,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:38,474 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:38,474 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:38,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:38,475 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:38,475 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:38,475 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:38,475 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:38,475 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:38,475 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:38,475 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:38,475 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:38,475 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:38,475 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:38,475 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:38,475 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:38,475 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:38,476 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:38,476 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:38,476 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:38,476 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. started... 2023-10-27 23:54:38,477 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9cee4e7411fb3bdf7d42508bea93aaf 2/2 column families, dataSize=120 B heapSize=928 B 2023-10-27 23:54:38,482 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,482 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,482 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,482 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,482 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:38,482 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:38,890 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/8c96dc4277614992b4b234ae26b7080b 2023-10-27 23:54:38,902 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/6f3d26c0ecaa48759ac32341fb3223bc 2023-10-27 23:54:38,905 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/8c96dc4277614992b4b234ae26b7080b as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/8c96dc4277614992b4b234ae26b7080b 2023-10-27 23:54:38,909 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/8c96dc4277614992b4b234ae26b7080b, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:54:38,909 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/6f3d26c0ecaa48759ac32341fb3223bc as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6f3d26c0ecaa48759ac32341fb3223bc 2023-10-27 23:54:38,912 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6f3d26c0ecaa48759ac32341fb3223bc, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:54:38,913 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for a9cee4e7411fb3bdf7d42508bea93aaf in 436ms, sequenceid=6, compaction requested=false 2023-10-27 23:54:38,913 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegion' 2023-10-27 23:54:38,913 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9cee4e7411fb3bdf7d42508bea93aaf: 2023-10-27 23:54:38,914 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:38,914 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:38,914 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:38,914 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:38,914 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:38,914 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:38,945 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,945 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:38,946 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,946 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:38,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-10-27 23:54:38,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@779be991[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:38,948 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-10-27 23:54:38,948 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,973 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,973 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,973 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,973 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:38,973 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:38,973 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:38,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,973 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:38,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,973 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:38,974 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:38,974 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:38,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:38,974 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:38,974 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:38,974 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:38,974 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:38,982 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:38,982 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:38,982 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:38,990 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:38,991 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:38,991 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:38,999 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:38,999 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:38,999 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:38,999 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:38,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:38,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:38,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:38,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:54:39,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-10-27 23:54:39,003 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-10-27 23:54:39,003 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:39,003 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:39,003 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-10-27 23:54:39,003 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:39,012 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,012 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,012 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,012 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,012 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:39,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,012 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:39,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,012 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:39,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,013 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,013 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:39,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,013 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,014 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:39,015 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:39,015 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,024 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:39,024 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:39,024 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,024 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:39,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-10-27 23:54:39,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:39,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:39,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:39,024 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:39,024 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:39,024 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:39,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:39,024 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:39,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:39,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:39,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:39,024 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:39,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:39,024 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:39,024 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:39,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:39,025 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,025 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:39,025 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:39,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:39,025 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,025 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,025 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,025 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,025 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,025 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,125 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:39,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:39,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:39,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:39,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-10-27 23:54:39,137 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-10-27 23:54:39,137 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:39,137 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-10-27 23:54:39,137 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-10-27 23:54:39,137 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,137 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,145 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,145 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:39,145 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:39,145 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:39,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:39,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:39,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:39,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:39,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:39,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:39,146 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,146 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:39,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:39,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:39,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:39,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:39,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:39,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:39,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:39,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:39,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:39,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:39,147 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:39,147 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:39,147 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:39,147 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:39,148 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:39,148 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:39,148 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:39,148 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:39,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:39,149 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:39,149 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:39,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:39,149 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:39,149 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:39,149 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. started... 2023-10-27 23:54:39,149 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:39,149 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:39,149 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9cee4e7411fb3bdf7d42508bea93aaf 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:39,149 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:39,149 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:39,157 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,165 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,165 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,165 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:39,165 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,166 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:39,566 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/3e1ad265bf04465995e50d1b36b7e846 2023-10-27 23:54:39,578 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/25ee8219411d4add8de1af5c999eed88 2023-10-27 23:54:39,581 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/3e1ad265bf04465995e50d1b36b7e846 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/3e1ad265bf04465995e50d1b36b7e846 2023-10-27 23:54:39,584 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/3e1ad265bf04465995e50d1b36b7e846, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:54:39,585 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/25ee8219411d4add8de1af5c999eed88 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/25ee8219411d4add8de1af5c999eed88 2023-10-27 23:54:39,588 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/25ee8219411d4add8de1af5c999eed88, entries=10, sequenceid=19, filesize=5.0 K 2023-10-27 23:54:39,589 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a9cee4e7411fb3bdf7d42508bea93aaf in 440ms, sequenceid=19, compaction requested=false 2023-10-27 23:54:39,589 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9cee4e7411fb3bdf7d42508bea93aaf: 2023-10-27 23:54:39,589 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:39,589 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:39,589 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:39,589 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,589 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:39,589 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:39,620 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,620 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:39,621 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,621 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:39,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-10-27 23:54:39,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1128714f[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:39,623 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-10-27 23:54:39,623 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,632 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,632 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,632 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,632 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:39,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,632 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:39,632 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:39,633 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:39,633 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:39,632 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,633 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,632 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:39,633 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:39,633 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:39,633 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:39,640 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:39,640 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:39,640 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:39,649 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:39,649 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,649 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:39,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,649 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:39,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:39,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:39,657 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:39,657 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:39,657 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:39,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:39,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:39,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:39,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:39,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:39,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:54:39,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-10-27 23:54:39,660 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-10-27 23:54:39,660 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:39,660 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:39,660 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-10-27 23:54:39,660 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:39,670 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,670 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,670 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,670 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:39,670 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:39,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,670 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:39,671 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:39,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:39,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:39,671 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,671 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:39,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,671 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:39,673 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:39,673 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,673 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:39,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:39,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:39,699 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:39,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-10-27 23:54:39,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:39,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:39,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:39,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:39,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:39,699 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:39,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:39,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:39,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:39,699 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:39,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:39,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:39,699 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:39,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:39,700 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:39,700 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:39,700 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:39,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:39,700 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:39,700 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:39,700 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,700 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,700 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,800 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:39,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:39,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:39,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:39,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-10-27 23:54:39,812 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-10-27 23:54:39,812 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:39,812 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-10-27 23:54:39,812 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-10-27 23:54:39,812 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,812 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:39,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:39,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:39,824 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:39,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:39,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:39,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:39,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:39,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:39,824 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:39,824 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:39,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:39,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:39,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:39,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:39,826 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:39,826 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:39,826 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. started... 2023-10-27 23:54:39,826 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9cee4e7411fb3bdf7d42508bea93aaf 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:39,832 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,840 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,840 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,840 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:39,841 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:39,841 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:40,242 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f 2023-10-27 23:54:40,252 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/16007d3a09474b43bc59a8b56c033d6d 2023-10-27 23:54:40,255 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f 2023-10-27 23:54:40,259 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:54:40,259 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/16007d3a09474b43bc59a8b56c033d6d as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/16007d3a09474b43bc59a8b56c033d6d 2023-10-27 23:54:40,262 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/16007d3a09474b43bc59a8b56c033d6d, entries=10, sequenceid=32, filesize=5.0 K 2023-10-27 23:54:40,263 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a9cee4e7411fb3bdf7d42508bea93aaf in 437ms, sequenceid=32, compaction requested=true 2023-10-27 23:54:40,263 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9cee4e7411fb3bdf7d42508bea93aaf: 2023-10-27 23:54:40,263 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:40,263 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:40,263 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:40,263 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,263 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:40,263 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:40,295 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,295 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:40,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:40,296 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,296 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:40,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:40,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:40,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:40,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-10-27 23:54:40,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8f1035[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:40,298 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-10-27 23:54:40,298 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,307 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,307 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,307 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,307 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:40,307 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:40,307 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:40,308 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:40,307 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,308 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:40,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,307 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:40,308 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:40,308 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:40,308 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:40,308 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:40,315 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:40,315 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:40,315 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:40,324 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:40,324 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,324 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:40,324 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:40,324 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:40,324 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:40,324 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:40,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:40,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:40,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:40,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:54:40,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-10-27 23:54:40,327 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-10-27 23:54:40,328 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:40,328 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:40,328 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-10-27 23:54:40,328 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:40,337 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,337 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,337 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,337 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,337 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,337 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,337 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,338 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:40,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:40,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,338 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,338 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:40,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,340 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:40,340 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,340 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-10-27 23:54:40,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:40,349 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:40,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:40,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,349 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,350 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,350 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,350 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,350 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,450 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:40,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:40,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:40,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:40,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-10-27 23:54:40,461 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-10-27 23:54:40,462 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:40,462 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-10-27 23:54:40,462 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-10-27 23:54:40,462 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,462 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,470 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,470 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,470 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,470 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,471 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,471 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:40,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:40,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:40,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:40,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:40,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:40,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:40,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:40,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:40,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:40,472 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:40,472 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:40,472 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. started... 2023-10-27 23:54:40,473 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9cee4e7411fb3bdf7d42508bea93aaf 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:40,482 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,484 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/4ec040ba90374d2cac76acce98c52fea 2023-10-27 23:54:40,491 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,491 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,491 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:40,491 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,491 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:40,501 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/478c04fde3324be8b677317f5b070900 2023-10-27 23:54:40,504 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/4ec040ba90374d2cac76acce98c52fea as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/4ec040ba90374d2cac76acce98c52fea 2023-10-27 23:54:40,508 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/4ec040ba90374d2cac76acce98c52fea, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:54:40,508 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/478c04fde3324be8b677317f5b070900 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/478c04fde3324be8b677317f5b070900 2023-10-27 23:54:40,511 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/478c04fde3324be8b677317f5b070900, entries=10, sequenceid=45, filesize=5.0 K 2023-10-27 23:54:40,512 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a9cee4e7411fb3bdf7d42508bea93aaf in 39ms, sequenceid=45, compaction requested=true 2023-10-27 23:54:40,512 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9cee4e7411fb3bdf7d42508bea93aaf: 2023-10-27 23:54:40,512 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:40,512 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:40,512 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:40,512 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,512 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:40,512 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:40,520 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,520 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:40,521 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,521 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-10-27 23:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b7f00d5[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:40,522 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-10-27 23:54:40,523 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,532 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,532 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,532 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,532 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,533 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:40,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,533 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,532 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:40,533 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:40,533 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:40,533 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:40,533 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:40,533 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:40,533 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:40,533 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:40,576 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:40,576 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:40,576 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:40,576 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,576 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:40,576 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:40,576 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:40,576 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:40,576 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:40,576 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:40,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:40,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:40,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:40,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:54:40,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-10-27 23:54:40,581 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-10-27 23:54:40,581 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:40,581 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:40,581 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-10-27 23:54:40,581 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:40,599 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,599 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,599 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,599 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,599 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,599 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:40,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:40,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:40,599 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,599 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,600 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:40,600 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,602 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,602 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,602 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,612 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,612 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,612 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,612 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,612 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-10-27 23:54:40,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:40,612 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,612 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,612 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,612 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:40,612 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,613 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,613 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:40,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,613 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:40,613 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,613 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,613 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,613 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,613 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,613 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,613 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,713 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:40,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:40,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:40,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:40,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-10-27 23:54:40,725 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-10-27 23:54:40,725 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:40,725 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-10-27 23:54:40,725 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-10-27 23:54:40,726 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,726 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,732 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,732 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,732 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,732 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,733 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,733 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:40,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:40,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:40,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:40,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:40,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:40,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:40,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:40,734 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:40,734 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:40,734 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. started... 2023-10-27 23:54:40,734 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9cee4e7411fb3bdf7d42508bea93aaf 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:40,740 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,744 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/b366093603bd46a2abfe0f651987b409 2023-10-27 23:54:40,749 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,749 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:40,749 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,749 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,749 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:40,755 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/ff12e1a362ee44c099ff285a0f3ab54d 2023-10-27 23:54:40,758 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/b366093603bd46a2abfe0f651987b409 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b366093603bd46a2abfe0f651987b409 2023-10-27 23:54:40,762 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b366093603bd46a2abfe0f651987b409, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:54:40,762 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/ff12e1a362ee44c099ff285a0f3ab54d as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/ff12e1a362ee44c099ff285a0f3ab54d 2023-10-27 23:54:40,766 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/ff12e1a362ee44c099ff285a0f3ab54d, entries=10, sequenceid=58, filesize=5.0 K 2023-10-27 23:54:40,766 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a9cee4e7411fb3bdf7d42508bea93aaf in 32ms, sequenceid=58, compaction requested=true 2023-10-27 23:54:40,766 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9cee4e7411fb3bdf7d42508bea93aaf: 2023-10-27 23:54:40,766 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:40,766 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:40,767 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:40,767 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,767 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:40,767 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:40,774 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,774 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:40,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:40,774 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,774 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:40,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:40,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:40,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-10-27 23:54:40,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5986badf[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:40,776 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-10-27 23:54:40,776 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,782 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,782 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,782 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,782 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,782 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:40,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,782 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:40,782 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:40,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,782 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:40,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,782 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:40,783 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:40,783 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:40,783 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:40,783 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:40,783 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:40,790 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:40,790 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,790 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,790 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:40,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:40,799 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:40,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:40,799 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:40,799 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:40,799 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:40,799 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:40,799 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:40,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:54:40,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-10-27 23:54:40,802 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-10-27 23:54:40,802 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:40,802 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:40,802 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-10-27 23:54:40,802 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:40,812 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,812 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,812 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,812 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,812 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,812 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,812 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,813 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,813 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:40,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:40,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,813 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:40,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,815 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,815 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:40,815 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,824 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:40,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-10-27 23:54:40,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:40,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:40,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,824 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,824 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:40,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:40,825 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:40,825 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:40,825 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,825 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,825 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,825 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,825 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,825 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,825 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,925 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:40,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:40,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(936): Client=jenkins//188.40.62.62 procedure request for: flush-table-proc 2023-10-27 23:54:40,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-10-27 23:54:40,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-10-27 23:54:40,937 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-10-27 23:54:40,937 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:40,937 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-10-27 23:54:40,937 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-10-27 23:54:40,938 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,938 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,945 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,945 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,945 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,945 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:40,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,945 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,946 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-10-27 23:54:40,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-10-27 23:54:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:40,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:40,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-10-27 23:54:40,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:40,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:40,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:40,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:40,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:40,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-10-27 23:54:40,947 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,39059,1698450767147' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:40,947 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. started... 2023-10-27 23:54:40,947 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:40,948 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,36599,1698450767241' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:40,948 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9cee4e7411fb3bdf7d42508bea93aaf 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-10-27 23:54:40,958 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/b3b9ed3e14794e62bc26c06411d04fa4 2023-10-27 23:54:40,968 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/928fa89349644c25b6a9aec2f6eab271 2023-10-27 23:54:40,972 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/b3b9ed3e14794e62bc26c06411d04fa4 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b3b9ed3e14794e62bc26c06411d04fa4 2023-10-27 23:54:40,975 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b3b9ed3e14794e62bc26c06411d04fa4, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:54:40,975 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,976 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/928fa89349644c25b6a9aec2f6eab271 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/928fa89349644c25b6a9aec2f6eab271 2023-10-27 23:54:40,979 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/928fa89349644c25b6a9aec2f6eab271, entries=10, sequenceid=71, filesize=5.0 K 2023-10-27 23:54:40,980 INFO [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a9cee4e7411fb3bdf7d42508bea93aaf in 32ms, sequenceid=71, compaction requested=true 2023-10-27 23:54:40,980 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9cee4e7411fb3bdf7d42508bea93aaf: 2023-10-27 23:54:40,980 DEBUG [rs(jenkins-hbase16.apache.org,43641,1698450767195)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:40,980 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-10-27 23:54:40,980 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-10-27 23:54:40,980 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,980 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-10-27 23:54:40,980 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-10-27 23:54:40,987 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,987 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,987 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:40,995 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,995 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,995 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,995 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:40,995 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:40,996 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-10-27 23:54:40,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:40,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:40,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:40,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:40,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:40,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:40,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:40,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase16.apache.org,43641,1698450767195' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-10-27 23:54:40,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7adc6092[Count = 0] remaining members to acquire global barrier 2023-10-27 23:54:40,998 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-10-27 23:54:40,998 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:41,007 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:41,007 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:41,007 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:41,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:41,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:41,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:41,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:41,007 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:41,007 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:41,007 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-10-27 23:54:41,007 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:41,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:41,008 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,36599,1698450767241' in zk 2023-10-27 23:54:41,008 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:41,008 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:41,008 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-10-27 23:54:41,008 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,39059,1698450767147' in zk 2023-10-27 23:54:41,008 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-10-27 23:54:41,008 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase16.apache.org,43641,1698450767195' in zk 2023-10-27 23:54:41,015 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:41,015 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:41,015 DEBUG [member: 'jenkins-hbase16.apache.org,36599,1698450767241' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:41,024 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:41,024 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:41,024 DEBUG [member: 'jenkins-hbase16.apache.org,39059,1698450767147' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:41,032 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-10-27 23:54:41,032 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:41,032 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,032 DEBUG [member: 'jenkins-hbase16.apache.org,43641,1698450767195' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-10-27 23:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase16.apache.org,43641,1698450767195': 2023-10-27 23:54:41,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase16.apache.org,43641,1698450767195' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-10-27 23:54:41,036 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-10-27 23:54:41,036 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-10-27 23:54:41,036 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-10-27 23:54:41,036 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-10-27 23:54:41,036 INFO [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-10-27 23:54:41,045 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,045 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,045 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,045 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,045 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:41,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-10-27 23:54:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-10-27 23:54:41,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,046 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,045 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-10-27 23:54:41,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:41,046 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:41,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,046 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:41,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:41,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:41,046 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-10-27 23:54:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:41,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:41,048 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-10-27 23:54:41,048 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:41,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-10-27 23:54:41,048 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:41,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:41,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:41,057 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:41,057 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:41,057 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:41,057 DEBUG [(jenkins-hbase16.apache.org,40039,1698450766041)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-10-27 23:54:41,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:41,057 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-10-27 23:54:41,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:41,057 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:41,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-10-27 23:54:41,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-10-27 23:54:41,057 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:41,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:41,057 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:41,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:41,058 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:41,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-10-27 23:54:41,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:41,057 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-10-27 23:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-10-27 23:54:41,058 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,058 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-10-27 23:54:41,058 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-10-27 23:54:41,058 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-10-27 23:54:41,058 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:41,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-10-27 23:54:41,058 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-10-27 23:54:41,058 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:41,058 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,058 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-10-27 23:54:41,058 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-10-27 23:54:41,158 DEBUG [Listener at localhost/43373] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-10-27 23:54:41,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-10-27 23:54:41,166 DEBUG [Listener at localhost/43373] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,167 DEBUG [Listener at localhost/43373] procedure2.ProcedureExecutor(1028): Stored pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=a9cee4e7411fb3bdf7d42508bea93aaf, daughterA=77cfdd0cb1a665ae153b2eda23085fa9, daughterB=9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:41,168 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=a9cee4e7411fb3bdf7d42508bea93aaf, daughterA=77cfdd0cb1a665ae153b2eda23085fa9, daughterB=9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:41,168 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=a9cee4e7411fb3bdf7d42508bea93aaf, daughterA=77cfdd0cb1a665ae153b2eda23085fa9, daughterB=9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:41,168 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=a9cee4e7411fb3bdf7d42508bea93aaf, daughterA=77cfdd0cb1a665ae153b2eda23085fa9, daughterB=9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:41,169 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=145, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=a9cee4e7411fb3bdf7d42508bea93aaf, UNASSIGN}] 2023-10-27 23:54:41,170 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=145, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=a9cee4e7411fb3bdf7d42508bea93aaf, UNASSIGN 2023-10-27 23:54:41,170 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=145 updating hbase:meta row=a9cee4e7411fb3bdf7d42508bea93aaf, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,170 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450881170"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450881170"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450881170"}]},"ts":"1698450881170"} 2023-10-27 23:54:41,172 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=146, ppid=145, state=RUNNABLE; CloseRegionProcedure a9cee4e7411fb3bdf7d42508bea93aaf, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:54:41,323 DEBUG [RSProcedureDispatcher-pool-4] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:41,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:41,324 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing a9cee4e7411fb3bdf7d42508bea93aaf, disabling compactions & flushes 2023-10-27 23:54:41,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:41,324 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:41,324 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. after waiting 0 ms 2023-10-27 23:54:41,324 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:41,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing a9cee4e7411fb3bdf7d42508bea93aaf 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-10-27 23:54:41,733 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/15f261fabdf74a8593c4effeba65835a 2023-10-27 23:54:42,145 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/6610cf34f9ea46588f34747499746f06 2023-10-27 23:54:42,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf1/15f261fabdf74a8593c4effeba65835a as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/15f261fabdf74a8593c4effeba65835a 2023-10-27 23:54:42,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/15f261fabdf74a8593c4effeba65835a, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:54:42,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/.tmp/cf2/6610cf34f9ea46588f34747499746f06 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6610cf34f9ea46588f34747499746f06 2023-10-27 23:54:42,157 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6610cf34f9ea46588f34747499746f06, entries=8, sequenceid=82, filesize=5.0 K 2023-10-27 23:54:42,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for a9cee4e7411fb3bdf7d42508bea93aaf in 834ms, sequenceid=82, compaction requested=true 2023-10-27 23:54:42,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-10-27 23:54:42,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:42,163 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. 2023-10-27 23:54:42,163 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for a9cee4e7411fb3bdf7d42508bea93aaf: 2023-10-27 23:54:42,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,164 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=145 updating hbase:meta row=a9cee4e7411fb3bdf7d42508bea93aaf, regionState=CLOSED 2023-10-27 23:54:42,164 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450882164"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450882164"}]},"ts":"1698450882164"} 2023-10-27 23:54:42,166 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=146, resume processing ppid=145 2023-10-27 23:54:42,166 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=146, ppid=145, state=SUCCESS; CloseRegionProcedure a9cee4e7411fb3bdf7d42508bea93aaf, server=jenkins-hbase16.apache.org,43641,1698450767195 in 993 msec 2023-10-27 23:54:42,167 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=145, resume processing ppid=144 2023-10-27 23:54:42,167 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=145, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=a9cee4e7411fb3bdf7d42508bea93aaf, UNASSIGN in 997 msec 2023-10-27 23:54:42,177 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=144 splitting 14 storefiles, region=a9cee4e7411fb3bdf7d42508bea93aaf, threads=14 2023-10-27 23:54:42,177 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/16007d3a09474b43bc59a8b56c033d6d for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,177 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/25ee8219411d4add8de1af5c999eed88 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,177 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/478c04fde3324be8b677317f5b070900 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,178 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6610cf34f9ea46588f34747499746f06 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,178 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6f3d26c0ecaa48759ac32341fb3223bc for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,178 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/928fa89349644c25b6a9aec2f6eab271 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,178 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/ff12e1a362ee44c099ff285a0f3ab54d for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,178 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/15f261fabdf74a8593c4effeba65835a for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,178 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/3e1ad265bf04465995e50d1b36b7e846 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,178 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/4ec040ba90374d2cac76acce98c52fea for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,178 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,178 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/8c96dc4277614992b4b234ae26b7080b for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,179 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b366093603bd46a2abfe0f651987b409 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,179 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b3b9ed3e14794e62bc26c06411d04fa4 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,337 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-27 23:54:42,352 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/15f261fabdf74a8593c4effeba65835a for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,352 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6610cf34f9ea46588f34747499746f06 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,356 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/25ee8219411d4add8de1af5c999eed88 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,356 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/4ec040ba90374d2cac76acce98c52fea for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,357 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/ff12e1a362ee44c099ff285a0f3ab54d for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,357 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/478c04fde3324be8b677317f5b070900 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,366 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b3b9ed3e14794e62bc26c06411d04fa4 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,371 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b366093603bd46a2abfe0f651987b409 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,395 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/928fa89349644c25b6a9aec2f6eab271 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,402 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,407 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6f3d26c0ecaa48759ac32341fb3223bc for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,731 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/16007d3a09474b43bc59a8b56c033d6d for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,786 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/3e1ad265bf04465995e50d1b36b7e846 for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,787 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/8c96dc4277614992b4b234ae26b7080b for region: a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:42,787 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=144 split storefiles for region a9cee4e7411fb3bdf7d42508bea93aaf Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-10-27 23:54:42,834 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-10-27 23:54:42,835 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-10-27 23:54:42,836 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1698450882836"},{"qualifier":"splitA","vlen":56,"tag":[],"timestamp":"1698450882836"},{"qualifier":"splitB","vlen":56,"tag":[],"timestamp":"1698450882836"}]},"ts":"1698450882836"} 2023-10-27 23:54:42,836 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450882836"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450882836"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450882836"}]},"ts":"1698450882836"} 2023-10-27 23:54:42,836 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450882836"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450882836"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450882836"}]},"ts":"1698450882836"} 2023-10-27 23:54:42,839 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=77cfdd0cb1a665ae153b2eda23085fa9, ASSIGN}, {pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=9cf0ed7ce82a5a965938a3c210f4c47a, ASSIGN}] 2023-10-27 23:54:42,839 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=9cf0ed7ce82a5a965938a3c210f4c47a, ASSIGN 2023-10-27 23:54:42,840 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=9cf0ed7ce82a5a965938a3c210f4c47a, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase16.apache.org,43641,1698450767195; forceNewPlan=false, retain=false 2023-10-27 23:54:42,840 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=77cfdd0cb1a665ae153b2eda23085fa9, ASSIGN 2023-10-27 23:54:42,841 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=77cfdd0cb1a665ae153b2eda23085fa9, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase16.apache.org,43641,1698450767195; forceNewPlan=false, retain=false 2023-10-27 23:54:42,990 INFO [jenkins-hbase16:40039] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-10-27 23:54:42,992 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=147 updating hbase:meta row=77cfdd0cb1a665ae153b2eda23085fa9, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:42,992 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450882992"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450882992"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450882992"}]},"ts":"1698450882992"} 2023-10-27 23:54:42,993 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=149, ppid=147, state=RUNNABLE; OpenRegionProcedure 77cfdd0cb1a665ae153b2eda23085fa9, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:54:42,995 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=148 updating hbase:meta row=9cf0ed7ce82a5a965938a3c210f4c47a, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:42,995 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450882995"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450882995"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450882995"}]},"ts":"1698450882995"} 2023-10-27 23:54:42,996 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=150, ppid=148, state=RUNNABLE; OpenRegionProcedure 9cf0ed7ce82a5a965938a3c210f4c47a, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:54:43,144 DEBUG [RSProcedureDispatcher-pool-5] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:43,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a. 2023-10-27 23:54:43,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9cf0ed7ce82a5a965938a3c210f4c47a, NAME => 'testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.', STARTKEY => '41', ENDKEY => ''} 2023-10-27 23:54:43,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:54:43,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion 9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:43,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:43,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:43,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:43,148 INFO [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:43,149 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1 2023-10-27 23:54:43,149 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1 2023-10-27 23:54:43,149 INFO [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9cf0ed7ce82a5a965938a3c210f4c47a columnFamilyName cf1 2023-10-27 23:54:43,162 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/15f261fabdf74a8593c4effeba65835a.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/15f261fabdf74a8593c4effeba65835a-top 2023-10-27 23:54:43,166 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/3e1ad265bf04465995e50d1b36b7e846.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/3e1ad265bf04465995e50d1b36b7e846-top 2023-10-27 23:54:43,169 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/4ec040ba90374d2cac76acce98c52fea.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/4ec040ba90374d2cac76acce98c52fea-top 2023-10-27 23:54:43,173 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f-top 2023-10-27 23:54:43,176 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/8c96dc4277614992b4b234ae26b7080b.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/8c96dc4277614992b4b234ae26b7080b-top 2023-10-27 23:54:43,180 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/b366093603bd46a2abfe0f651987b409.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b366093603bd46a2abfe0f651987b409-top 2023-10-27 23:54:43,183 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/b3b9ed3e14794e62bc26c06411d04fa4.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b3b9ed3e14794e62bc26c06411d04fa4-top 2023-10-27 23:54:43,183 INFO [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(310): Store=9cf0ed7ce82a5a965938a3c210f4c47a/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:43,183 INFO [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:43,184 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2 2023-10-27 23:54:43,184 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2 2023-10-27 23:54:43,184 INFO [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9cf0ed7ce82a5a965938a3c210f4c47a columnFamilyName cf2 2023-10-27 23:54:43,237 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/16007d3a09474b43bc59a8b56c033d6d.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/16007d3a09474b43bc59a8b56c033d6d-top 2023-10-27 23:54:43,257 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/25ee8219411d4add8de1af5c999eed88.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/25ee8219411d4add8de1af5c999eed88-top 2023-10-27 23:54:43,261 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/478c04fde3324be8b677317f5b070900.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/478c04fde3324be8b677317f5b070900-top 2023-10-27 23:54:43,282 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/6610cf34f9ea46588f34747499746f06.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6610cf34f9ea46588f34747499746f06-top 2023-10-27 23:54:43,299 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/6f3d26c0ecaa48759ac32341fb3223bc.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6f3d26c0ecaa48759ac32341fb3223bc-top 2023-10-27 23:54:43,309 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/928fa89349644c25b6a9aec2f6eab271.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/928fa89349644c25b6a9aec2f6eab271-top 2023-10-27 23:54:43,314 DEBUG [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/ff12e1a362ee44c099ff285a0f3ab54d.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/ff12e1a362ee44c099ff285a0f3ab54d-top 2023-10-27 23:54:43,314 INFO [StoreOpener-9cf0ed7ce82a5a965938a3c210f4c47a-1] regionserver.HStore(310): Store=9cf0ed7ce82a5a965938a3c210f4c47a/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:43,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:43,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:43,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:54:43,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:43,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 9cf0ed7ce82a5a965938a3c210f4c47a; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63992792, jitterRate=-0.046433091163635254}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:54:43,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 9cf0ed7ce82a5a965938a3c210f4c47a: 2023-10-27 23:54:43,326 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a., pid=150, masterSystemTime=1698450883144 2023-10-27 23:54:43,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:43,327 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:54:43,327 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:54:43,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:43,327 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a. 2023-10-27 23:54:43,327 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a. 2023-10-27 23:54:43,327 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1912): 9cf0ed7ce82a5a965938a3c210f4c47a/cf2 is initiating minor compaction (all files) 2023-10-27 23:54:43,328 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1912): 9cf0ed7ce82a5a965938a3c210f4c47a/cf1 is initiating minor compaction (all files) 2023-10-27 23:54:43,328 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9cf0ed7ce82a5a965938a3c210f4c47a/cf2 in testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a. 2023-10-27 23:54:43,328 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9cf0ed7ce82a5a965938a3c210f4c47a/cf1 in testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a. 2023-10-27 23:54:43,328 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/6f3d26c0ecaa48759ac32341fb3223bc.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6f3d26c0ecaa48759ac32341fb3223bc-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/25ee8219411d4add8de1af5c999eed88.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/25ee8219411d4add8de1af5c999eed88-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/16007d3a09474b43bc59a8b56c033d6d.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/16007d3a09474b43bc59a8b56c033d6d-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/478c04fde3324be8b677317f5b070900.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/478c04fde3324be8b677317f5b070900-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/ff12e1a362ee44c099ff285a0f3ab54d.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/ff12e1a362ee44c099ff285a0f3ab54d-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/928fa89349644c25b6a9aec2f6eab271.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/928fa89349644c25b6a9aec2f6eab271-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/6610cf34f9ea46588f34747499746f06.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6610cf34f9ea46588f34747499746f06-top] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/.tmp, totalSize=34.9 K 2023-10-27 23:54:43,328 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/8c96dc4277614992b4b234ae26b7080b.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/8c96dc4277614992b4b234ae26b7080b-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/3e1ad265bf04465995e50d1b36b7e846.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/3e1ad265bf04465995e50d1b36b7e846-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/4ec040ba90374d2cac76acce98c52fea.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/4ec040ba90374d2cac76acce98c52fea-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/b366093603bd46a2abfe0f651987b409.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b366093603bd46a2abfe0f651987b409-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/b3b9ed3e14794e62bc26c06411d04fa4.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b3b9ed3e14794e62bc26c06411d04fa4-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/15f261fabdf74a8593c4effeba65835a.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/15f261fabdf74a8593c4effeba65835a-top] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/.tmp, totalSize=34.9 K 2023-10-27 23:54:43,329 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 6f3d26c0ecaa48759ac32341fb3223bc.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1698450878458 2023-10-27 23:54:43,329 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 8c96dc4277614992b4b234ae26b7080b.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1698450878458 2023-10-27 23:54:43,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a. 2023-10-27 23:54:43,329 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a. 2023-10-27 23:54:43,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9. 2023-10-27 23:54:43,330 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 25ee8219411d4add8de1af5c999eed88.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1698450879127 2023-10-27 23:54:43,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 77cfdd0cb1a665ae153b2eda23085fa9, NAME => 'testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.', STARTKEY => '', ENDKEY => '41'} 2023-10-27 23:54:43,330 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=148 updating hbase:meta row=9cf0ed7ce82a5a965938a3c210f4c47a, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:43,330 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 3e1ad265bf04465995e50d1b36b7e846.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1698450879127 2023-10-27 23:54:43,330 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450883330"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450883330"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450883330"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450883330"}]},"ts":"1698450883330"} 2023-10-27 23:54:43,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-10-27 23:54:43,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion 77cfdd0cb1a665ae153b2eda23085fa9 2023-10-27 23:54:43,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-10-27 23:54:43,330 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 16007d3a09474b43bc59a8b56c033d6d.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1698450879801 2023-10-27 23:54:43,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 77cfdd0cb1a665ae153b2eda23085fa9 2023-10-27 23:54:43,330 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 5edd0d7870bb473b9fd4824e2b2e3a8f.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1698450879801 2023-10-27 23:54:43,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 77cfdd0cb1a665ae153b2eda23085fa9 2023-10-27 23:54:43,331 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 478c04fde3324be8b677317f5b070900.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1698450880451 2023-10-27 23:54:43,331 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 4ec040ba90374d2cac76acce98c52fea.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1698450880451 2023-10-27 23:54:43,331 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting ff12e1a362ee44c099ff285a0f3ab54d.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1698450880715 2023-10-27 23:54:43,331 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting b366093603bd46a2abfe0f651987b409.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1698450880715 2023-10-27 23:54:43,332 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 928fa89349644c25b6a9aec2f6eab271.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1698450880927 2023-10-27 23:54:43,332 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting b3b9ed3e14794e62bc26c06411d04fa4.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1698450880927 2023-10-27 23:54:43,332 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 6610cf34f9ea46588f34747499746f06.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1698450881160 2023-10-27 23:54:43,333 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 15f261fabdf74a8593c4effeba65835a.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1698450881160 2023-10-27 23:54:43,333 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=150, resume processing ppid=148 2023-10-27 23:54:43,333 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=150, ppid=148, state=SUCCESS; OpenRegionProcedure 9cf0ed7ce82a5a965938a3c210f4c47a, server=jenkins-hbase16.apache.org,43641,1698450767195 in 335 msec 2023-10-27 23:54:43,335 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=148, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=9cf0ed7ce82a5a965938a3c210f4c47a, ASSIGN in 494 msec 2023-10-27 23:54:43,343 INFO [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 77cfdd0cb1a665ae153b2eda23085fa9 2023-10-27 23:54:43,347 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9cf0ed7ce82a5a965938a3c210f4c47a#cf1#compaction#142 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:54:43,347 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1 2023-10-27 23:54:43,348 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1 2023-10-27 23:54:43,348 INFO [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 77cfdd0cb1a665ae153b2eda23085fa9 columnFamilyName cf1 2023-10-27 23:54:43,355 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] throttle.PressureAwareThroughputController(145): 9cf0ed7ce82a5a965938a3c210f4c47a#cf2#compaction#143 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:54:43,379 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/15f261fabdf74a8593c4effeba65835a.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/15f261fabdf74a8593c4effeba65835a-bottom 2023-10-27 23:54:43,382 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/3e1ad265bf04465995e50d1b36b7e846.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/3e1ad265bf04465995e50d1b36b7e846-bottom 2023-10-27 23:54:43,393 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/4ec040ba90374d2cac76acce98c52fea.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/4ec040ba90374d2cac76acce98c52fea-bottom 2023-10-27 23:54:43,395 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/.tmp/cf1/405e64df03dd46f89169ba08ab38a09d as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/405e64df03dd46f89169ba08ab38a09d 2023-10-27 23:54:43,396 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f-bottom 2023-10-27 23:54:43,398 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/.tmp/cf2/3c84210695ad48409029bdc5a69ee7f0 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/3c84210695ad48409029bdc5a69ee7f0 2023-10-27 23:54:43,400 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/8c96dc4277614992b4b234ae26b7080b.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/8c96dc4277614992b4b234ae26b7080b-bottom 2023-10-27 23:54:43,400 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9cf0ed7ce82a5a965938a3c210f4c47a/cf1 of 9cf0ed7ce82a5a965938a3c210f4c47a into 405e64df03dd46f89169ba08ab38a09d(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:54:43,401 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9cf0ed7ce82a5a965938a3c210f4c47a: 2023-10-27 23:54:43,401 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a., storeName=9cf0ed7ce82a5a965938a3c210f4c47a/cf1, priority=9, startTime=1698450883326; duration=0sec 2023-10-27 23:54:43,401 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:43,403 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9cf0ed7ce82a5a965938a3c210f4c47a/cf2 of 9cf0ed7ce82a5a965938a3c210f4c47a into 3c84210695ad48409029bdc5a69ee7f0(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:54:43,403 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9cf0ed7ce82a5a965938a3c210f4c47a: 2023-10-27 23:54:43,403 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a., storeName=9cf0ed7ce82a5a965938a3c210f4c47a/cf2, priority=9, startTime=1698450883327; duration=0sec 2023-10-27 23:54:43,403 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:43,404 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/b366093603bd46a2abfe0f651987b409.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b366093603bd46a2abfe0f651987b409-bottom 2023-10-27 23:54:43,408 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/b3b9ed3e14794e62bc26c06411d04fa4.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b3b9ed3e14794e62bc26c06411d04fa4-bottom 2023-10-27 23:54:43,408 INFO [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(310): Store=77cfdd0cb1a665ae153b2eda23085fa9/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:43,408 INFO [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 77cfdd0cb1a665ae153b2eda23085fa9 2023-10-27 23:54:43,409 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2 2023-10-27 23:54:43,409 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2 2023-10-27 23:54:43,409 INFO [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 77cfdd0cb1a665ae153b2eda23085fa9 columnFamilyName cf2 2023-10-27 23:54:43,421 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/16007d3a09474b43bc59a8b56c033d6d.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/16007d3a09474b43bc59a8b56c033d6d-bottom 2023-10-27 23:54:43,428 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/25ee8219411d4add8de1af5c999eed88.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/25ee8219411d4add8de1af5c999eed88-bottom 2023-10-27 23:54:43,431 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/478c04fde3324be8b677317f5b070900.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/478c04fde3324be8b677317f5b070900-bottom 2023-10-27 23:54:43,435 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/6610cf34f9ea46588f34747499746f06.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6610cf34f9ea46588f34747499746f06-bottom 2023-10-27 23:54:43,437 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/6f3d26c0ecaa48759ac32341fb3223bc.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6f3d26c0ecaa48759ac32341fb3223bc-bottom 2023-10-27 23:54:43,440 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/928fa89349644c25b6a9aec2f6eab271.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/928fa89349644c25b6a9aec2f6eab271-bottom 2023-10-27 23:54:43,443 DEBUG [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(539): loaded hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/ff12e1a362ee44c099ff285a0f3ab54d.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/ff12e1a362ee44c099ff285a0f3ab54d-bottom 2023-10-27 23:54:43,443 INFO [StoreOpener-77cfdd0cb1a665ae153b2eda23085fa9-1] regionserver.HStore(310): Store=77cfdd0cb1a665ae153b2eda23085fa9/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-10-27 23:54:43,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9 2023-10-27 23:54:43,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9 2023-10-27 23:54:43,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-10-27 23:54:43,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 77cfdd0cb1a665ae153b2eda23085fa9 2023-10-27 23:54:43,447 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 77cfdd0cb1a665ae153b2eda23085fa9; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66389092, jitterRate=-0.010725438594818115}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-10-27 23:54:43,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 77cfdd0cb1a665ae153b2eda23085fa9: 2023-10-27 23:54:43,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9., pid=149, masterSystemTime=1698450883144 2023-10-27 23:54:43,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:54:43,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-10-27 23:54:43,448 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:54:43,448 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-10-27 23:54:43,449 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9. 2023-10-27 23:54:43,449 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9. 2023-10-27 23:54:43,449 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1912): 77cfdd0cb1a665ae153b2eda23085fa9/cf1 is initiating minor compaction (all files) 2023-10-27 23:54:43,449 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1912): 77cfdd0cb1a665ae153b2eda23085fa9/cf2 is initiating minor compaction (all files) 2023-10-27 23:54:43,449 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 77cfdd0cb1a665ae153b2eda23085fa9/cf1 in testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9. 2023-10-27 23:54:43,449 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 77cfdd0cb1a665ae153b2eda23085fa9/cf2 in testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9. 2023-10-27 23:54:43,449 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/8c96dc4277614992b4b234ae26b7080b.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/8c96dc4277614992b4b234ae26b7080b-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/3e1ad265bf04465995e50d1b36b7e846.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/3e1ad265bf04465995e50d1b36b7e846-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/4ec040ba90374d2cac76acce98c52fea.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/4ec040ba90374d2cac76acce98c52fea-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/b366093603bd46a2abfe0f651987b409.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b366093603bd46a2abfe0f651987b409-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/b3b9ed3e14794e62bc26c06411d04fa4.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b3b9ed3e14794e62bc26c06411d04fa4-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/15f261fabdf74a8593c4effeba65835a.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/15f261fabdf74a8593c4effeba65835a-bottom] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/.tmp, totalSize=34.9 K 2023-10-27 23:54:43,449 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/6f3d26c0ecaa48759ac32341fb3223bc.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6f3d26c0ecaa48759ac32341fb3223bc-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/25ee8219411d4add8de1af5c999eed88.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/25ee8219411d4add8de1af5c999eed88-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/16007d3a09474b43bc59a8b56c033d6d.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/16007d3a09474b43bc59a8b56c033d6d-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/478c04fde3324be8b677317f5b070900.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/478c04fde3324be8b677317f5b070900-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/ff12e1a362ee44c099ff285a0f3ab54d.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/ff12e1a362ee44c099ff285a0f3ab54d-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/928fa89349644c25b6a9aec2f6eab271.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/928fa89349644c25b6a9aec2f6eab271-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/6610cf34f9ea46588f34747499746f06.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6610cf34f9ea46588f34747499746f06-bottom] into tmpdir=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/.tmp, totalSize=34.9 K 2023-10-27 23:54:43,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9. 2023-10-27 23:54:43,450 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9. 2023-10-27 23:54:43,450 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=147 updating hbase:meta row=77cfdd0cb1a665ae153b2eda23085fa9, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:43,450 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 8c96dc4277614992b4b234ae26b7080b.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450878458 2023-10-27 23:54:43,450 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 6f3d26c0ecaa48759ac32341fb3223bc.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1698450878458 2023-10-27 23:54:43,450 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450883450"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1698450883450"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1698450883450"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1698450883450"}]},"ts":"1698450883450"} 2023-10-27 23:54:43,450 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 3e1ad265bf04465995e50d1b36b7e846.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450879127 2023-10-27 23:54:43,451 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 25ee8219411d4add8de1af5c999eed88.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1698450879127 2023-10-27 23:54:43,451 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 5edd0d7870bb473b9fd4824e2b2e3a8f.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450879801 2023-10-27 23:54:43,451 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 16007d3a09474b43bc59a8b56c033d6d.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1698450879801 2023-10-27 23:54:43,451 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 4ec040ba90374d2cac76acce98c52fea.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450880451 2023-10-27 23:54:43,451 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 478c04fde3324be8b677317f5b070900.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1698450880451 2023-10-27 23:54:43,451 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting b366093603bd46a2abfe0f651987b409.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450880715 2023-10-27 23:54:43,452 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting ff12e1a362ee44c099ff285a0f3ab54d.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1698450880715 2023-10-27 23:54:43,452 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting b3b9ed3e14794e62bc26c06411d04fa4.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450880927 2023-10-27 23:54:43,452 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 928fa89349644c25b6a9aec2f6eab271.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1698450880927 2023-10-27 23:54:43,452 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] compactions.Compactor(207): Compacting 6610cf34f9ea46588f34747499746f06.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1698450881160 2023-10-27 23:54:43,452 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] compactions.Compactor(207): Compacting 15f261fabdf74a8593c4effeba65835a.a9cee4e7411fb3bdf7d42508bea93aaf, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1698450881160 2023-10-27 23:54:43,452 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=149, resume processing ppid=147 2023-10-27 23:54:43,452 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=149, ppid=147, state=SUCCESS; OpenRegionProcedure 77cfdd0cb1a665ae153b2eda23085fa9, server=jenkins-hbase16.apache.org,43641,1698450767195 in 458 msec 2023-10-27 23:54:43,454 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=147, resume processing ppid=144 2023-10-27 23:54:43,454 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=147, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=77cfdd0cb1a665ae153b2eda23085fa9, ASSIGN in 613 msec 2023-10-27 23:54:43,455 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=144, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegion, parent=a9cee4e7411fb3bdf7d42508bea93aaf, daughterA=77cfdd0cb1a665ae153b2eda23085fa9, daughterB=9cf0ed7ce82a5a965938a3c210f4c47a in 2.2880 sec 2023-10-27 23:54:43,459 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] throttle.PressureAwareThroughputController(145): 77cfdd0cb1a665ae153b2eda23085fa9#cf2#compaction#144 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:54:43,471 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] throttle.PressureAwareThroughputController(145): 77cfdd0cb1a665ae153b2eda23085fa9#cf1#compaction#145 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-10-27 23:54:43,478 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/.tmp/cf2/4c098c105c044db8af0af94de24a61d3 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/4c098c105c044db8af0af94de24a61d3 2023-10-27 23:54:43,482 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 77cfdd0cb1a665ae153b2eda23085fa9/cf2 of 77cfdd0cb1a665ae153b2eda23085fa9 into 4c098c105c044db8af0af94de24a61d3(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:54:43,482 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 77cfdd0cb1a665ae153b2eda23085fa9: 2023-10-27 23:54:43,482 INFO [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9., storeName=77cfdd0cb1a665ae153b2eda23085fa9/cf2, priority=9, startTime=1698450883448; duration=0sec 2023-10-27 23:54:43,482 DEBUG [RS:1;jenkins-hbase16:43641-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:43,488 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/.tmp/cf1/b18289a13f7b4890895dd2d56f62fb56 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/b18289a13f7b4890895dd2d56f62fb56 2023-10-27 23:54:43,498 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 77cfdd0cb1a665ae153b2eda23085fa9/cf1 of 77cfdd0cb1a665ae153b2eda23085fa9 into b18289a13f7b4890895dd2d56f62fb56(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-10-27 23:54:43,498 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 77cfdd0cb1a665ae153b2eda23085fa9: 2023-10-27 23:54:43,498 INFO [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9., storeName=77cfdd0cb1a665ae153b2eda23085fa9/cf1, priority=9, startTime=1698450883448; duration=0sec 2023-10-27 23:54:43,498 DEBUG [RS:1;jenkins-hbase16:43641-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-10-27 23:54:43,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43641] ipc.CallRunner(144): callId: 1128 service: ClientService methodName: Scan size: 143 connection: 188.40.62.62:48696 deadline: 1698450943668, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf. is not online on jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:43,779 DEBUG [Listener at localhost/43373] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-10-27 23:54:43,786 INFO [Listener at localhost/43373] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-10-27 23:54:43,787 DEBUG [Listener at localhost/43373] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-10-27 23:54:43,795 WARN [Listener at localhost/43373] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-10-27 23:54:43,796 INFO [Listener at localhost/43373] client.HBaseAdmin$15(890): Started disable of testSplitTableRegion 2023-10-27 23:54:43,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$11(2418): Client=jenkins//188.40.62.62 disable testSplitTableRegion 2023-10-27 23:54:43,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=151, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegion 2023-10-27 23:54:43,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=151 2023-10-27 23:54:43,800 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450883799"}]},"ts":"1698450883799"} 2023-10-27 23:54:43,801 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=DISABLING in hbase:meta 2023-10-27 23:54:43,832 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegion to state=DISABLING 2023-10-27 23:54:43,833 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=152, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=77cfdd0cb1a665ae153b2eda23085fa9, UNASSIGN}, {pid=153, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=9cf0ed7ce82a5a965938a3c210f4c47a, UNASSIGN}] 2023-10-27 23:54:43,833 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=9cf0ed7ce82a5a965938a3c210f4c47a, UNASSIGN 2023-10-27 23:54:43,834 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=9cf0ed7ce82a5a965938a3c210f4c47a, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:43,834 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450883834"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450883834"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450883834"}]},"ts":"1698450883834"} 2023-10-27 23:54:43,835 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=154, ppid=153, state=RUNNABLE; CloseRegionProcedure 9cf0ed7ce82a5a965938a3c210f4c47a, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:54:43,836 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=77cfdd0cb1a665ae153b2eda23085fa9, UNASSIGN 2023-10-27 23:54:43,837 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=77cfdd0cb1a665ae153b2eda23085fa9, regionState=CLOSING, regionLocation=jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:43,837 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450883837"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1698450883837"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1698450883837"}]},"ts":"1698450883837"} 2023-10-27 23:54:43,840 INFO [PEWorker-1] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=155, ppid=152, state=RUNNABLE; CloseRegionProcedure 77cfdd0cb1a665ae153b2eda23085fa9, server=jenkins-hbase16.apache.org,43641,1698450767195}] 2023-10-27 23:54:43,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=151 2023-10-27 23:54:43,986 DEBUG [RSProcedureDispatcher-pool-3] master.ServerManager(702): New admin connection to jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:43,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:43,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 9cf0ed7ce82a5a965938a3c210f4c47a, disabling compactions & flushes 2023-10-27 23:54:43,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a. 2023-10-27 23:54:43,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a. 2023-10-27 23:54:43,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a. after waiting 0 ms 2023-10-27 23:54:43,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a. 2023-10-27 23:54:43,987 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/8c96dc4277614992b4b234ae26b7080b.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/8c96dc4277614992b4b234ae26b7080b-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/3e1ad265bf04465995e50d1b36b7e846.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/3e1ad265bf04465995e50d1b36b7e846-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/4ec040ba90374d2cac76acce98c52fea.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/4ec040ba90374d2cac76acce98c52fea-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/b366093603bd46a2abfe0f651987b409.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b366093603bd46a2abfe0f651987b409-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/b3b9ed3e14794e62bc26c06411d04fa4.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b3b9ed3e14794e62bc26c06411d04fa4-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/15f261fabdf74a8593c4effeba65835a.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/15f261fabdf74a8593c4effeba65835a-top] to archive 2023-10-27 23:54:43,988 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:54:43,989 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/8c96dc4277614992b4b234ae26b7080b.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/8c96dc4277614992b4b234ae26b7080b.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:43,990 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/3e1ad265bf04465995e50d1b36b7e846.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/3e1ad265bf04465995e50d1b36b7e846.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:43,991 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:43,992 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/4ec040ba90374d2cac76acce98c52fea.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/4ec040ba90374d2cac76acce98c52fea.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:43,993 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/b366093603bd46a2abfe0f651987b409.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/b366093603bd46a2abfe0f651987b409.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:43,994 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/b3b9ed3e14794e62bc26c06411d04fa4.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/b3b9ed3e14794e62bc26c06411d04fa4.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:43,995 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/15f261fabdf74a8593c4effeba65835a.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/15f261fabdf74a8593c4effeba65835a.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:43,996 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/6f3d26c0ecaa48759ac32341fb3223bc.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6f3d26c0ecaa48759ac32341fb3223bc-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/25ee8219411d4add8de1af5c999eed88.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/25ee8219411d4add8de1af5c999eed88-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/16007d3a09474b43bc59a8b56c033d6d.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/16007d3a09474b43bc59a8b56c033d6d-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/478c04fde3324be8b677317f5b070900.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/478c04fde3324be8b677317f5b070900-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/ff12e1a362ee44c099ff285a0f3ab54d.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/ff12e1a362ee44c099ff285a0f3ab54d-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/928fa89349644c25b6a9aec2f6eab271.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/928fa89349644c25b6a9aec2f6eab271-top, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/6610cf34f9ea46588f34747499746f06.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6610cf34f9ea46588f34747499746f06-top] to archive 2023-10-27 23:54:43,997 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:54:43,998 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/6f3d26c0ecaa48759ac32341fb3223bc.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/6f3d26c0ecaa48759ac32341fb3223bc.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:43,999 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/25ee8219411d4add8de1af5c999eed88.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/25ee8219411d4add8de1af5c999eed88.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:43,999 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/16007d3a09474b43bc59a8b56c033d6d.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/16007d3a09474b43bc59a8b56c033d6d.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,000 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/478c04fde3324be8b677317f5b070900.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/478c04fde3324be8b677317f5b070900.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,001 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/ff12e1a362ee44c099ff285a0f3ab54d.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/ff12e1a362ee44c099ff285a0f3ab54d.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,002 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/928fa89349644c25b6a9aec2f6eab271.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/928fa89349644c25b6a9aec2f6eab271.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,003 DEBUG [StoreCloser-testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/6610cf34f9ea46588f34747499746f06.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/6610cf34f9ea46588f34747499746f06.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-10-27 23:54:44,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:44,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a. 2023-10-27 23:54:44,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 9cf0ed7ce82a5a965938a3c210f4c47a: 2023-10-27 23:54:44,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:44,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(111): Close 77cfdd0cb1a665ae153b2eda23085fa9 2023-10-27 23:54:44,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 77cfdd0cb1a665ae153b2eda23085fa9, disabling compactions & flushes 2023-10-27 23:54:44,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9. 2023-10-27 23:54:44,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9. 2023-10-27 23:54:44,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9. after waiting 0 ms 2023-10-27 23:54:44,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9. 2023-10-27 23:54:44,008 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=9cf0ed7ce82a5a965938a3c210f4c47a, regionState=CLOSED 2023-10-27 23:54:44,008 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450884008"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450884008"}]},"ts":"1698450884008"} 2023-10-27 23:54:44,009 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/8c96dc4277614992b4b234ae26b7080b.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/8c96dc4277614992b4b234ae26b7080b-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/3e1ad265bf04465995e50d1b36b7e846.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/3e1ad265bf04465995e50d1b36b7e846-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/4ec040ba90374d2cac76acce98c52fea.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/4ec040ba90374d2cac76acce98c52fea-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/b366093603bd46a2abfe0f651987b409.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b366093603bd46a2abfe0f651987b409-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/b3b9ed3e14794e62bc26c06411d04fa4.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b3b9ed3e14794e62bc26c06411d04fa4-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/15f261fabdf74a8593c4effeba65835a.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/15f261fabdf74a8593c4effeba65835a-bottom] to archive 2023-10-27 23:54:44,010 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:54:44,010 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=154, resume processing ppid=153 2023-10-27 23:54:44,011 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=154, ppid=153, state=SUCCESS; CloseRegionProcedure 9cf0ed7ce82a5a965938a3c210f4c47a, server=jenkins-hbase16.apache.org,43641,1698450767195 in 174 msec 2023-10-27 23:54:44,011 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/8c96dc4277614992b4b234ae26b7080b.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/8c96dc4277614992b4b234ae26b7080b.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,011 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=153, ppid=151, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=9cf0ed7ce82a5a965938a3c210f4c47a, UNASSIGN in 178 msec 2023-10-27 23:54:44,012 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/3e1ad265bf04465995e50d1b36b7e846.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/3e1ad265bf04465995e50d1b36b7e846.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,013 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,014 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/4ec040ba90374d2cac76acce98c52fea.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/4ec040ba90374d2cac76acce98c52fea.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,015 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/b366093603bd46a2abfe0f651987b409.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/b366093603bd46a2abfe0f651987b409.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,015 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/b3b9ed3e14794e62bc26c06411d04fa4.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/b3b9ed3e14794e62bc26c06411d04fa4.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,016 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/15f261fabdf74a8593c4effeba65835a.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/15f261fabdf74a8593c4effeba65835a.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,018 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/6f3d26c0ecaa48759ac32341fb3223bc.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6f3d26c0ecaa48759ac32341fb3223bc-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/25ee8219411d4add8de1af5c999eed88.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/25ee8219411d4add8de1af5c999eed88-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/16007d3a09474b43bc59a8b56c033d6d.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/16007d3a09474b43bc59a8b56c033d6d-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/478c04fde3324be8b677317f5b070900.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/478c04fde3324be8b677317f5b070900-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/ff12e1a362ee44c099ff285a0f3ab54d.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/ff12e1a362ee44c099ff285a0f3ab54d-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/928fa89349644c25b6a9aec2f6eab271.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/928fa89349644c25b6a9aec2f6eab271-bottom, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/6610cf34f9ea46588f34747499746f06.a9cee4e7411fb3bdf7d42508bea93aaf->hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6610cf34f9ea46588f34747499746f06-bottom] to archive 2023-10-27 23:54:44,018 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-10-27 23:54:44,019 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/6f3d26c0ecaa48759ac32341fb3223bc.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/6f3d26c0ecaa48759ac32341fb3223bc.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,020 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/25ee8219411d4add8de1af5c999eed88.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/25ee8219411d4add8de1af5c999eed88.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,021 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/16007d3a09474b43bc59a8b56c033d6d.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/16007d3a09474b43bc59a8b56c033d6d.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,022 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/478c04fde3324be8b677317f5b070900.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/478c04fde3324be8b677317f5b070900.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,023 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/ff12e1a362ee44c099ff285a0f3ab54d.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/ff12e1a362ee44c099ff285a0f3ab54d.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,024 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/928fa89349644c25b6a9aec2f6eab271.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/928fa89349644c25b6a9aec2f6eab271.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,024 DEBUG [StoreCloser-testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/6610cf34f9ea46588f34747499746f06.a9cee4e7411fb3bdf7d42508bea93aaf to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/6610cf34f9ea46588f34747499746f06.a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-10-27 23:54:44,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:44,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9. 2023-10-27 23:54:44,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 77cfdd0cb1a665ae153b2eda23085fa9: 2023-10-27 23:54:44,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.UnassignRegionHandler(149): Closed 77cfdd0cb1a665ae153b2eda23085fa9 2023-10-27 23:54:44,030 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=77cfdd0cb1a665ae153b2eda23085fa9, regionState=CLOSED 2023-10-27 23:54:44,030 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1698450884030"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1698450884030"}]},"ts":"1698450884030"} 2023-10-27 23:54:44,032 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=155, resume processing ppid=152 2023-10-27 23:54:44,032 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=155, ppid=152, state=SUCCESS; CloseRegionProcedure 77cfdd0cb1a665ae153b2eda23085fa9, server=jenkins-hbase16.apache.org,43641,1698450767195 in 193 msec 2023-10-27 23:54:44,033 INFO [PEWorker-1] procedure2.ProcedureExecutor(1823): Finished subprocedure pid=152, resume processing ppid=151 2023-10-27 23:54:44,033 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=152, ppid=151, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=77cfdd0cb1a665ae153b2eda23085fa9, UNASSIGN in 199 msec 2023-10-27 23:54:44,034 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1698450884034"}]},"ts":"1698450884034"} 2023-10-27 23:54:44,034 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=DISABLED in hbase:meta 2023-10-27 23:54:44,045 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegion to state=DISABLED 2023-10-27 23:54:44,047 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=151, state=SUCCESS; DisableTableProcedure table=testSplitTableRegion in 249 msec 2023-10-27 23:54:44,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=151 2023-10-27 23:54:44,102 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegion, procId: 151 completed 2023-10-27 23:54:44,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.HMaster$5(2228): Client=jenkins//188.40.62.62 delete testSplitTableRegion 2023-10-27 23:54:44,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] procedure2.ProcedureExecutor(1028): Stored pid=156, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegion 2023-10-27 23:54:44,104 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=156, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-10-27 23:54:44,105 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=156, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-10-27 23:54:44,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=156 2023-10-27 23:54:44,108 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,108 DEBUG [HFileArchiver-14] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9 2023-10-27 23:54:44,108 DEBUG [HFileArchiver-15] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:44,110 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/recovered.edits] 2023-10-27 23:54:44,110 DEBUG [HFileArchiver-15] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/recovered.edits] 2023-10-27 23:54:44,110 DEBUG [HFileArchiver-14] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2, FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/recovered.edits] 2023-10-27 23:54:44,113 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/405e64df03dd46f89169ba08ab38a09d to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf1/405e64df03dd46f89169ba08ab38a09d 2023-10-27 23:54:44,114 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/b18289a13f7b4890895dd2d56f62fb56 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf1/b18289a13f7b4890895dd2d56f62fb56 2023-10-27 23:54:44,114 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/15f261fabdf74a8593c4effeba65835a to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/15f261fabdf74a8593c4effeba65835a 2023-10-27 23:54:44,115 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/3e1ad265bf04465995e50d1b36b7e846 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/3e1ad265bf04465995e50d1b36b7e846 2023-10-27 23:54:44,115 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/3c84210695ad48409029bdc5a69ee7f0 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/cf2/3c84210695ad48409029bdc5a69ee7f0 2023-10-27 23:54:44,116 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/4c098c105c044db8af0af94de24a61d3 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/cf2/4c098c105c044db8af0af94de24a61d3 2023-10-27 23:54:44,116 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/4ec040ba90374d2cac76acce98c52fea to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/4ec040ba90374d2cac76acce98c52fea 2023-10-27 23:54:44,117 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/5edd0d7870bb473b9fd4824e2b2e3a8f 2023-10-27 23:54:44,118 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/recovered.edits/90.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a/recovered.edits/90.seqid 2023-10-27 23:54:44,118 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/recovered.edits/90.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9/recovered.edits/90.seqid 2023-10-27 23:54:44,119 DEBUG [HFileArchiver-15] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/9cf0ed7ce82a5a965938a3c210f4c47a 2023-10-27 23:54:44,119 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/8c96dc4277614992b4b234ae26b7080b to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/8c96dc4277614992b4b234ae26b7080b 2023-10-27 23:54:44,119 DEBUG [HFileArchiver-14] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/77cfdd0cb1a665ae153b2eda23085fa9 2023-10-27 23:54:44,120 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b366093603bd46a2abfe0f651987b409 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b366093603bd46a2abfe0f651987b409 2023-10-27 23:54:44,120 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b3b9ed3e14794e62bc26c06411d04fa4 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf1/b3b9ed3e14794e62bc26c06411d04fa4 2023-10-27 23:54:44,122 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/16007d3a09474b43bc59a8b56c033d6d to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/16007d3a09474b43bc59a8b56c033d6d 2023-10-27 23:54:44,123 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/25ee8219411d4add8de1af5c999eed88 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/25ee8219411d4add8de1af5c999eed88 2023-10-27 23:54:44,124 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/478c04fde3324be8b677317f5b070900 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/478c04fde3324be8b677317f5b070900 2023-10-27 23:54:44,125 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6610cf34f9ea46588f34747499746f06 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6610cf34f9ea46588f34747499746f06 2023-10-27 23:54:44,126 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6f3d26c0ecaa48759ac32341fb3223bc to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/6f3d26c0ecaa48759ac32341fb3223bc 2023-10-27 23:54:44,127 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/928fa89349644c25b6a9aec2f6eab271 to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/928fa89349644c25b6a9aec2f6eab271 2023-10-27 23:54:44,128 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/ff12e1a362ee44c099ff285a0f3ab54d to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/cf2/ff12e1a362ee44c099ff285a0f3ab54d 2023-10-27 23:54:44,130 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/recovered.edits/85.seqid to hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/archive/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf/recovered.edits/85.seqid 2023-10-27 23:54:44,131 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/.tmp/data/default/testSplitTableRegion/a9cee4e7411fb3bdf7d42508bea93aaf 2023-10-27 23:54:44,131 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegion regions 2023-10-27 23:54:44,132 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=156, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-10-27 23:54:44,134 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegion from hbase:meta 2023-10-27 23:54:44,136 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegion' descriptor. 2023-10-27 23:54:44,137 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=156, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-10-27 23:54:44,137 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegion' from region states. 2023-10-27 23:54:44,137 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450884137"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:44,137 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450884137"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:44,138 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1698450884137"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:44,139 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-10-27 23:54:44,139 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a9cee4e7411fb3bdf7d42508bea93aaf, NAME => 'testSplitTableRegion,,1698450877955.a9cee4e7411fb3bdf7d42508bea93aaf.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 77cfdd0cb1a665ae153b2eda23085fa9, NAME => 'testSplitTableRegion,,1698450881166.77cfdd0cb1a665ae153b2eda23085fa9.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => 9cf0ed7ce82a5a965938a3c210f4c47a, NAME => 'testSplitTableRegion,41,1698450881166.9cf0ed7ce82a5a965938a3c210f4c47a.', STARTKEY => '41', ENDKEY => ''}] 2023-10-27 23:54:44,139 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegion' as deleted. 2023-10-27 23:54:44,139 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1698450884139"}]},"ts":"9223372036854775807"} 2023-10-27 23:54:44,141 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegion state from META 2023-10-27 23:54:44,149 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=156, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-10-27 23:54:44,150 INFO [PEWorker-1] procedure2.ProcedureExecutor(1409): Finished pid=156, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegion in 47 msec 2023-10-27 23:54:44,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40039] master.MasterRpcServices(1230): Checking to see if procedure is done pid=156 2023-10-27 23:54:44,206 INFO [Listener at localhost/43373] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegion, procId: 156 completed 2023-10-27 23:54:44,218 INFO [Listener at localhost/43373] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegion Thread=379 (was 379), OpenFileDescriptor=838 (was 841), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 214), ProcessCount=170 (was 170), AvailableMemoryMB=5829 (was 5395) - AvailableMemoryMB LEAK? - 2023-10-27 23:54:44,218 INFO [Listener at localhost/43373] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-10-27 23:54:44,218 INFO [Listener at localhost/43373] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-10-27 23:54:44,219 DEBUG [Listener at localhost/43373] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28683c56 to 127.0.0.1:58988 2023-10-27 23:54:44,219 DEBUG [Listener at localhost/43373] ipc.AbstractRpcClient(494): Stopping rpc client 2023-10-27 23:54:44,219 DEBUG [Listener at localhost/43373] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-10-27 23:54:44,220 DEBUG [Listener at localhost/43373] util.JVMClusterUtil(257): Found active master hash=1132304223, stopped=false 2023-10-27 23:54:44,220 INFO [Listener at localhost/43373] master.ServerManager(888): Cluster shutdown requested of master=jenkins-hbase16.apache.org,40039,1698450766041 2023-10-27 23:54:44,228 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-10-27 23:54:44,229 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-10-27 23:54:44,229 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-10-27 23:54:44,229 INFO [Listener at localhost/43373] procedure2.ProcedureExecutor(628): Stopping 2023-10-27 23:54:44,228 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-10-27 23:54:44,228 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-10-27 23:54:44,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-10-27 23:54:44,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-10-27 23:54:44,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-10-27 23:54:44,229 DEBUG [Listener at localhost/43373] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d742742 to 127.0.0.1:58988 2023-10-27 23:54:44,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-10-27 23:54:44,229 DEBUG [Listener at localhost/43373] ipc.AbstractRpcClient(494): Stopping rpc client 2023-10-27 23:54:44,230 INFO [Listener at localhost/43373] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase16.apache.org,39059,1698450767147' ***** 2023-10-27 23:54:44,230 INFO [Listener at localhost/43373] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-10-27 23:54:44,230 INFO [RS:0;jenkins-hbase16:39059] regionserver.HeapMemoryManager(220): Stopping 2023-10-27 23:54:44,230 INFO [RS:0;jenkins-hbase16:39059] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-10-27 23:54:44,230 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-10-27 23:54:44,230 INFO [RS:0;jenkins-hbase16:39059] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-10-27 23:54:44,230 INFO [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(3305): Received CLOSE for 5005e44f9940b2c7d862a285ee6d8687 2023-10-27 23:54:44,231 INFO [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(1144): stopping server jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:44,231 DEBUG [RS:0;jenkins-hbase16:39059] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x023f0099 to 127.0.0.1:58988 2023-10-27 23:54:44,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 5005e44f9940b2c7d862a285ee6d8687, disabling compactions & flushes 2023-10-27 23:54:44,231 DEBUG [RS:0;jenkins-hbase16:39059] ipc.AbstractRpcClient(494): Stopping rpc client 2023-10-27 23:54:44,231 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. 2023-10-27 23:54:44,231 INFO [Listener at localhost/43373] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase16.apache.org,43641,1698450767195' ***** 2023-10-27 23:54:44,231 INFO [Listener at localhost/43373] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-10-27 23:54:44,231 INFO [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-10-27 23:54:44,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. 2023-10-27 23:54:44,231 INFO [RS:1;jenkins-hbase16:43641] regionserver.HeapMemoryManager(220): Stopping 2023-10-27 23:54:44,231 INFO [RS:1;jenkins-hbase16:43641] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-10-27 23:54:44,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. after waiting 0 ms 2023-10-27 23:54:44,231 DEBUG [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(1478): Online Regions={5005e44f9940b2c7d862a285ee6d8687=hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687.} 2023-10-27 23:54:44,231 INFO [Listener at localhost/43373] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase16.apache.org,36599,1698450767241' ***** 2023-10-27 23:54:44,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. 2023-10-27 23:54:44,231 INFO [RS:1;jenkins-hbase16:43641] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-10-27 23:54:44,231 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-10-27 23:54:44,231 INFO [RS:1;jenkins-hbase16:43641] regionserver.HRegionServer(1144): stopping server jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:44,231 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 5005e44f9940b2c7d862a285ee6d8687 1/1 column families, dataSize=78 B heapSize=488 B 2023-10-27 23:54:44,231 INFO [Listener at localhost/43373] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-10-27 23:54:44,232 DEBUG [RS:1;jenkins-hbase16:43641] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c00dde6 to 127.0.0.1:58988 2023-10-27 23:54:44,232 DEBUG [RS:1;jenkins-hbase16:43641] ipc.AbstractRpcClient(494): Stopping rpc client 2023-10-27 23:54:44,232 INFO [RS:2;jenkins-hbase16:36599] regionserver.HeapMemoryManager(220): Stopping 2023-10-27 23:54:44,232 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-10-27 23:54:44,232 INFO [RS:1;jenkins-hbase16:43641] regionserver.HRegionServer(1170): stopping server jenkins-hbase16.apache.org,43641,1698450767195; all regions closed. 2023-10-27 23:54:44,232 DEBUG [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(1504): Waiting on 5005e44f9940b2c7d862a285ee6d8687 2023-10-27 23:54:44,232 INFO [RS:2;jenkins-hbase16:36599] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-10-27 23:54:44,232 INFO [RS:2;jenkins-hbase16:36599] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-10-27 23:54:44,232 INFO [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(1144): stopping server jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:44,232 DEBUG [RS:2;jenkins-hbase16:36599] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4980ca39 to 127.0.0.1:58988 2023-10-27 23:54:44,232 DEBUG [RS:2;jenkins-hbase16:36599] ipc.AbstractRpcClient(494): Stopping rpc client 2023-10-27 23:54:44,232 INFO [RS:2;jenkins-hbase16:36599] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-10-27 23:54:44,232 INFO [RS:2;jenkins-hbase16:36599] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-10-27 23:54:44,232 INFO [RS:2;jenkins-hbase16:36599] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-10-27 23:54:44,232 INFO [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-10-27 23:54:44,233 INFO [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-10-27 23:54:44,233 DEBUG [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-10-27 23:54:44,233 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-10-27 23:54:44,233 DEBUG [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-10-27 23:54:44,233 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-10-27 23:54:44,233 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-10-27 23:54:44,233 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-10-27 23:54:44,233 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-10-27 23:54:44,233 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=79.49 KB heapSize=125.82 KB 2023-10-27 23:54:44,244 DEBUG [RS:1;jenkins-hbase16:43641] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/oldWALs 2023-10-27 23:54:44,244 INFO [RS:1;jenkins-hbase16:43641] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C43641%2C1698450767195:(num 1698450768913) 2023-10-27 23:54:44,244 DEBUG [RS:1;jenkins-hbase16:43641] ipc.AbstractRpcClient(494): Stopping rpc client 2023-10-27 23:54:44,244 INFO [RS:1;jenkins-hbase16:43641] regionserver.LeaseManager(133): Closed leases 2023-10-27 23:54:44,245 INFO [RS:1;jenkins-hbase16:43641] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-10-27 23:54:44,246 INFO [RS:1;jenkins-hbase16:43641] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-10-27 23:54:44,246 INFO [RS:1;jenkins-hbase16:43641] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-10-27 23:54:44,246 INFO [RS:1;jenkins-hbase16:43641] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-10-27 23:54:44,246 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-10-27 23:54:44,264 INFO [RS:1;jenkins-hbase16:43641] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:43641 2023-10-27 23:54:44,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/namespace/5005e44f9940b2c7d862a285ee6d8687/.tmp/info/9025d5605a6c4eae98395cbef672adbb 2023-10-27 23:54:44,294 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/namespace/5005e44f9940b2c7d862a285ee6d8687/.tmp/info/9025d5605a6c4eae98395cbef672adbb as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/namespace/5005e44f9940b2c7d862a285ee6d8687/info/9025d5605a6c4eae98395cbef672adbb 2023-10-27 23:54:44,295 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:44,295 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:44,295 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-10-27 23:54:44,295 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-10-27 23:54:44,295 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-10-27 23:54:44,295 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,43641,1698450767195 2023-10-27 23:54:44,296 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-10-27 23:54:44,296 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,43641,1698450767195] 2023-10-27 23:54:44,296 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,43641,1698450767195; numProcessing=1 2023-10-27 23:54:44,300 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/namespace/5005e44f9940b2c7d862a285ee6d8687/info/9025d5605a6c4eae98395cbef672adbb, entries=2, sequenceid=6, filesize=4.8 K 2023-10-27 23:54:44,300 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-10-27 23:54:44,300 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-10-27 23:54:44,300 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-10-27 23:54:44,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 5005e44f9940b2c7d862a285ee6d8687 in 70ms, sequenceid=6, compaction requested=false 2023-10-27 23:54:44,302 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=71.37 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/.tmp/info/89ff17adcfc2492e879cbe95d3b5f2c2 2023-10-27 23:54:44,307 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 89ff17adcfc2492e879cbe95d3b5f2c2 2023-10-27 23:54:44,312 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,43641,1698450767195 already deleted, retry=false 2023-10-27 23:54:44,312 INFO [RegionServerTracker-0] master.ServerManager(554): Cluster shutdown set; jenkins-hbase16.apache.org,43641,1698450767195 expired; onlineServers=2 2023-10-27 23:54:44,321 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/namespace/5005e44f9940b2c7d862a285ee6d8687/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-10-27 23:54:44,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:44,323 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. 2023-10-27 23:54:44,323 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 5005e44f9940b2c7d862a285ee6d8687: 2023-10-27 23:54:44,323 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1698450769316.5005e44f9940b2c7d862a285ee6d8687. 2023-10-27 23:54:44,330 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.58 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/.tmp/rep_barrier/b1a5637ad84a4e199eb14b11719a9e3b 2023-10-27 23:54:44,334 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b1a5637ad84a4e199eb14b11719a9e3b 2023-10-27 23:54:44,360 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=5.53 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/.tmp/table/510df3c73ee54850ad098f9ae75ffd91 2023-10-27 23:54:44,365 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 510df3c73ee54850ad098f9ae75ffd91 2023-10-27 23:54:44,365 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/.tmp/info/89ff17adcfc2492e879cbe95d3b5f2c2 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/info/89ff17adcfc2492e879cbe95d3b5f2c2 2023-10-27 23:54:44,369 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 89ff17adcfc2492e879cbe95d3b5f2c2 2023-10-27 23:54:44,369 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/info/89ff17adcfc2492e879cbe95d3b5f2c2, entries=59, sequenceid=210, filesize=11.1 K 2023-10-27 23:54:44,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/.tmp/rep_barrier/b1a5637ad84a4e199eb14b11719a9e3b as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/rep_barrier/b1a5637ad84a4e199eb14b11719a9e3b 2023-10-27 23:54:44,374 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b1a5637ad84a4e199eb14b11719a9e3b 2023-10-27 23:54:44,375 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/rep_barrier/b1a5637ad84a4e199eb14b11719a9e3b, entries=24, sequenceid=210, filesize=7.5 K 2023-10-27 23:54:44,376 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/.tmp/table/510df3c73ee54850ad098f9ae75ffd91 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/table/510df3c73ee54850ad098f9ae75ffd91 2023-10-27 23:54:44,380 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 510df3c73ee54850ad098f9ae75ffd91 2023-10-27 23:54:44,380 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/table/510df3c73ee54850ad098f9ae75ffd91, entries=36, sequenceid=210, filesize=8.0 K 2023-10-27 23:54:44,381 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~79.49 KB/81396, heapSize ~125.77 KB/128792, currentSize=0 B/0 for 1588230740 in 148ms, sequenceid=210, compaction requested=false 2023-10-27 23:54:44,391 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/data/hbase/meta/1588230740/recovered.edits/213.seqid, newMaxSeqId=213, maxSeqId=1 2023-10-27 23:54:44,391 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-10-27 23:54:44,391 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-10-27 23:54:44,392 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-10-27 23:54:44,392 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-10-27 23:54:44,393 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-10-27 23:54:44,404 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-10-27 23:54:44,404 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:43641-0x1018a4ce86c0002, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-10-27 23:54:44,404 INFO [RS:1;jenkins-hbase16:43641] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase16.apache.org,43641,1698450767195; zookeeper connection closed. 2023-10-27 23:54:44,405 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2db8944c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2db8944c 2023-10-27 23:54:44,432 INFO [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(1170): stopping server jenkins-hbase16.apache.org,39059,1698450767147; all regions closed. 2023-10-27 23:54:44,433 INFO [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(1170): stopping server jenkins-hbase16.apache.org,36599,1698450767241; all regions closed. 2023-10-27 23:54:44,442 DEBUG [RS:0;jenkins-hbase16:39059] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/oldWALs 2023-10-27 23:54:44,442 INFO [RS:0;jenkins-hbase16:39059] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C39059%2C1698450767147:(num 1698450768912) 2023-10-27 23:54:44,442 DEBUG [RS:0;jenkins-hbase16:39059] ipc.AbstractRpcClient(494): Stopping rpc client 2023-10-27 23:54:44,442 INFO [RS:0;jenkins-hbase16:39059] regionserver.LeaseManager(133): Closed leases 2023-10-27 23:54:44,442 INFO [RS:0;jenkins-hbase16:39059] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-10-27 23:54:44,443 INFO [RS:0;jenkins-hbase16:39059] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-10-27 23:54:44,443 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-10-27 23:54:44,443 INFO [RS:0;jenkins-hbase16:39059] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-10-27 23:54:44,443 INFO [RS:0;jenkins-hbase16:39059] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-10-27 23:54:44,444 INFO [RS:0;jenkins-hbase16:39059] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:39059 2023-10-27 23:54:44,445 DEBUG [RS:2;jenkins-hbase16:36599] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/oldWALs 2023-10-27 23:54:44,445 INFO [RS:2;jenkins-hbase16:36599] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C36599%2C1698450767241.meta:.meta(num 1698450769021) 2023-10-27 23:54:44,453 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:44,453 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-10-27 23:54:44,453 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,39059,1698450767147 2023-10-27 23:54:44,454 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,39059,1698450767147] 2023-10-27 23:54:44,454 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,39059,1698450767147; numProcessing=2 2023-10-27 23:54:44,460 DEBUG [RS:2;jenkins-hbase16:36599] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/oldWALs 2023-10-27 23:54:44,461 INFO [RS:2;jenkins-hbase16:36599] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C36599%2C1698450767241:(num 1698450768913) 2023-10-27 23:54:44,461 DEBUG [RS:2;jenkins-hbase16:36599] ipc.AbstractRpcClient(494): Stopping rpc client 2023-10-27 23:54:44,461 INFO [RS:2;jenkins-hbase16:36599] regionserver.LeaseManager(133): Closed leases 2023-10-27 23:54:44,461 INFO [RS:2;jenkins-hbase16:36599] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-10-27 23:54:44,462 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-10-27 23:54:44,463 INFO [RS:2;jenkins-hbase16:36599] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:36599 2023-10-27 23:54:44,470 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,39059,1698450767147 already deleted, retry=false 2023-10-27 23:54:44,470 INFO [RegionServerTracker-0] master.ServerManager(554): Cluster shutdown set; jenkins-hbase16.apache.org,39059,1698450767147 expired; onlineServers=1 2023-10-27 23:54:44,478 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,36599,1698450767241 2023-10-27 23:54:44,478 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-10-27 23:54:44,564 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-10-27 23:54:44,564 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:39059-0x1018a4ce86c0001, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-10-27 23:54:44,564 INFO [RS:0;jenkins-hbase16:39059] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase16.apache.org,39059,1698450767147; zookeeper connection closed. 2023-10-27 23:54:44,564 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,36599,1698450767241] 2023-10-27 23:54:44,564 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@29a147a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@29a147a 2023-10-27 23:54:44,564 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,36599,1698450767241; numProcessing=3 2023-10-27 23:54:44,582 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,36599,1698450767241 already deleted, retry=false 2023-10-27 23:54:44,582 INFO [RegionServerTracker-0] master.ServerManager(554): Cluster shutdown set; jenkins-hbase16.apache.org,36599,1698450767241 expired; onlineServers=0 2023-10-27 23:54:44,582 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase16.apache.org,40039,1698450766041' ***** 2023-10-27 23:54:44,582 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-10-27 23:54:44,583 DEBUG [M:0;jenkins-hbase16:40039] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2409d64, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-10-27 23:54:44,583 INFO [M:0;jenkins-hbase16:40039] regionserver.HRegionServer(1144): stopping server jenkins-hbase16.apache.org,40039,1698450766041 2023-10-27 23:54:44,583 INFO [M:0;jenkins-hbase16:40039] regionserver.HRegionServer(1170): stopping server jenkins-hbase16.apache.org,40039,1698450766041; all regions closed. 2023-10-27 23:54:44,583 DEBUG [M:0;jenkins-hbase16:40039] ipc.AbstractRpcClient(494): Stopping rpc client 2023-10-27 23:54:44,583 DEBUG [M:0;jenkins-hbase16:40039] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-10-27 23:54:44,584 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-10-27 23:54:44,584 DEBUG [M:0;jenkins-hbase16:40039] cleaner.HFileCleaner(317): Stopping file delete threads 2023-10-27 23:54:44,584 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1698450768440] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1698450768440,5,FailOnTimeoutGroup] 2023-10-27 23:54:44,584 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1698450768440] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1698450768440,5,FailOnTimeoutGroup] 2023-10-27 23:54:44,584 INFO [M:0;jenkins-hbase16:40039] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-10-27 23:54:44,584 INFO [M:0;jenkins-hbase16:40039] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-10-27 23:54:44,584 INFO [M:0;jenkins-hbase16:40039] hbase.ChoreService(369): Chore service for: master/jenkins-hbase16:0 had [] on shutdown 2023-10-27 23:54:44,584 DEBUG [M:0;jenkins-hbase16:40039] master.HMaster(1512): Stopping service threads 2023-10-27 23:54:44,584 INFO [M:0;jenkins-hbase16:40039] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-10-27 23:54:44,584 ERROR [M:0;jenkins-hbase16:40039] procedure2.ProcedureExecutor(652): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] Thread[HFileArchiver-14,5,PEWorkerGroup] Thread[HFileArchiver-15,5,PEWorkerGroup] Thread[HFileArchiver-16,5,PEWorkerGroup] 2023-10-27 23:54:44,584 INFO [M:0;jenkins-hbase16:40039] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-10-27 23:54:44,585 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-10-27 23:54:44,590 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-10-27 23:54:44,590 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-10-27 23:54:44,590 DEBUG [M:0;jenkins-hbase16:40039] zookeeper.ZKUtil(398): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-10-27 23:54:44,590 WARN [M:0;jenkins-hbase16:40039] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-10-27 23:54:44,591 INFO [M:0;jenkins-hbase16:40039] assignment.AssignmentManager(315): Stopping assignment manager 2023-10-27 23:54:44,591 INFO [M:0;jenkins-hbase16:40039] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-10-27 23:54:44,591 DEBUG [M:0;jenkins-hbase16:40039] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-10-27 23:54:44,591 INFO [M:0;jenkins-hbase16:40039] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-10-27 23:54:44,591 DEBUG [M:0;jenkins-hbase16:40039] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-10-27 23:54:44,591 DEBUG [M:0;jenkins-hbase16:40039] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-10-27 23:54:44,591 DEBUG [M:0;jenkins-hbase16:40039] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-10-27 23:54:44,591 INFO [M:0;jenkins-hbase16:40039] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=608.82 KB heapSize=719.72 KB 2023-10-27 23:54:44,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-10-27 23:54:44,627 INFO [M:0;jenkins-hbase16:40039] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=608.82 KB at sequenceid=1282 (bloomFilter=true), to=hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/1075287ef442443d98102d472f665423 2023-10-27 23:54:44,642 INFO [M:0;jenkins-hbase16:40039] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1075287ef442443d98102d472f665423 2023-10-27 23:54:44,643 DEBUG [M:0;jenkins-hbase16:40039] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/1075287ef442443d98102d472f665423 as hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/1075287ef442443d98102d472f665423 2023-10-27 23:54:44,647 INFO [M:0;jenkins-hbase16:40039] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1075287ef442443d98102d472f665423 2023-10-27 23:54:44,647 INFO [M:0;jenkins-hbase16:40039] regionserver.HStore(1080): Added hdfs://localhost:46525/user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/1075287ef442443d98102d472f665423, entries=156, sequenceid=1282, filesize=34.9 K 2023-10-27 23:54:44,650 INFO [M:0;jenkins-hbase16:40039] regionserver.HRegion(2948): Finished flush of dataSize ~608.82 KB/623427, heapSize ~719.70 KB/736976, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 59ms, sequenceid=1282, compaction requested=false 2023-10-27 23:54:44,651 INFO [M:0;jenkins-hbase16:40039] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-10-27 23:54:44,651 DEBUG [M:0;jenkins-hbase16:40039] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-10-27 23:54:44,659 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/4336e3e8-7792-f2ac-80ca-f86eded4c6c5/MasterData/WALs/jenkins-hbase16.apache.org,40039,1698450766041/jenkins-hbase16.apache.org%2C40039%2C1698450766041.1698450767722 not finished, retry = 0 2023-10-27 23:54:44,674 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-10-27 23:54:44,674 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): regionserver:36599-0x1018a4ce86c0003, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-10-27 23:54:44,674 INFO [RS:2;jenkins-hbase16:36599] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase16.apache.org,36599,1698450767241; zookeeper connection closed. 2023-10-27 23:54:44,683 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@48a2cce3] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@48a2cce3 2023-10-27 23:54:44,683 INFO [Listener at localhost/43373] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-10-27 23:54:44,760 INFO [M:0;jenkins-hbase16:40039] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-10-27 23:54:44,760 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-10-27 23:54:44,761 INFO [M:0;jenkins-hbase16:40039] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:40039 2023-10-27 23:54:44,770 DEBUG [M:0;jenkins-hbase16:40039] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase16.apache.org,40039,1698450766041 already deleted, retry=false 2023-10-27 23:54:44,879 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-10-27 23:54:44,879 INFO [M:0;jenkins-hbase16:40039] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase16.apache.org,40039,1698450766041; zookeeper connection closed. 2023-10-27 23:54:44,879 DEBUG [Listener at localhost/43373-EventThread] zookeeper.ZKWatcher(600): master:40039-0x1018a4ce86c0000, quorum=127.0.0.1:58988, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-10-27 23:54:44,881 WARN [Listener at localhost/43373] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-10-27 23:54:44,911 INFO [Listener at localhost/43373] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-10-27 23:54:45,017 WARN [BP-446492043-188.40.62.62-1698450761445 heartbeating to localhost/127.0.0.1:46525] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-10-27 23:54:45,017 WARN [BP-446492043-188.40.62.62-1698450761445 heartbeating to localhost/127.0.0.1:46525] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-446492043-188.40.62.62-1698450761445 (Datanode Uuid 9d32eec7-ccca-436f-a2e6-4184d95cd961) service to localhost/127.0.0.1:46525 2023-10-27 23:54:45,019 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/dfs/data/data5/current/BP-446492043-188.40.62.62-1698450761445] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-10-27 23:54:45,019 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/dfs/data/data6/current/BP-446492043-188.40.62.62-1698450761445] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-10-27 23:54:45,022 WARN [Listener at localhost/43373] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-10-27 23:54:45,027 INFO [Listener at localhost/43373] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-10-27 23:54:45,132 WARN [BP-446492043-188.40.62.62-1698450761445 heartbeating to localhost/127.0.0.1:46525] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-10-27 23:54:45,132 WARN [BP-446492043-188.40.62.62-1698450761445 heartbeating to localhost/127.0.0.1:46525] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-446492043-188.40.62.62-1698450761445 (Datanode Uuid 41efbad2-7a69-4304-ba15-0342fa6d22c3) service to localhost/127.0.0.1:46525 2023-10-27 23:54:45,133 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/dfs/data/data3/current/BP-446492043-188.40.62.62-1698450761445] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-10-27 23:54:45,133 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/dfs/data/data4/current/BP-446492043-188.40.62.62-1698450761445] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-10-27 23:54:45,135 WARN [Listener at localhost/43373] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-10-27 23:54:45,143 INFO [Listener at localhost/43373] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-10-27 23:54:45,248 WARN [BP-446492043-188.40.62.62-1698450761445 heartbeating to localhost/127.0.0.1:46525] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-10-27 23:54:45,248 WARN [BP-446492043-188.40.62.62-1698450761445 heartbeating to localhost/127.0.0.1:46525] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-446492043-188.40.62.62-1698450761445 (Datanode Uuid 36ff42f7-94a3-4a82-a7a9-0f471d56a50f) service to localhost/127.0.0.1:46525 2023-10-27 23:54:45,249 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/dfs/data/data1/current/BP-446492043-188.40.62.62-1698450761445] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-10-27 23:54:45,250 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/1c8920bb-9eb8-4522-8fd3-61d25b84ee21/cluster_8b2920e8-4b55-e229-de47-b283228c2957/dfs/data/data2/current/BP-446492043-188.40.62.62-1698450761445] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-10-27 23:54:45,272 INFO [Listener at localhost/43373] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-10-27 23:54:45,470 INFO [Listener at localhost/43373] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-10-27 23:54:45,754 INFO [Listener at localhost/43373] hbase.HBaseTestingUtility(1293): Minicluster is down